【Azure 应用服务】Azure Function App使用SendGrid发送邮件遇见异常消息The operation was canceled,分析源码渐入最源端

简介: 【Azure 应用服务】Azure Function App使用SendGrid发送邮件遇见异常消息The operation was canceled,分析源码渐入最源端

问题描述

在使用Azure Function App的 SendGrid Binging 功能,调用SendGrid服务器发送邮件功能时,遇到见间歇性,偶发性的异常。在重新运行SendGrid的Function,却又能恢复运行。

所以本文基于Azure Function使用SendGrid的异常错误日志,一步一步,分析源码中的方法内容。 然后调查为什么 Azure Function 没有自动重试(Retry)?

(如需要参考如何使用Azure Function SendGrid,参考:Azure Functions SendGrid 绑定)

错误消息:

System.Threading.Tasks.TaskCanceledException : The operation was canceled.
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async System.Net.Http.ConnectHelper.ConnectAsync(String host,Int32 port,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at async System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request,Boolean allowHttp2,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at async System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at async System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at async System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request,Boolean doRequestAuth,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async SendGrid.Helpers.Reliability.RetryDelegatingHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask,HttpRequestMessage request,CancellationTokenSource cts,Boolean disposeCts)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async SendGrid.SendGridClient.MakeRequest(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async SendGrid.SendGridClient.RequestAsync(Method method,String requestBody,String queryParams,String urlPath,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async SendGrid.SendGridClient.SendEmailAsync(SendGridMessage msg,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Client.SendGridClient.SendMessageAsync(SendGridMessage msg,CancellationToken cancellationToken) at C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.SendGrid\Client\SendGridClient.cs : 23
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Extensions.Bindings.SendGridMessageAsyncCollector.FlushAsync(CancellationToken cancellationToken) at C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.SendGrid\Bindings\SendGridMessageAsyncCollector.cs : 56
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Bindings.AsyncCollectorValueProvider`2.SetValueAsync[TUser,TMessage](Object value,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Bindings\AsyncCollector\AsyncCollectorValueProvider.cs : 49
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Binder.Complete(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Bindings\Runtime\Binder.cs : 150
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Bindings.Runtime.RuntimeValueProvider.SetValueAsync(Object value,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Bindings\Runtime\RuntimeValueProvider.cs : 34
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ParameterHelper.ProcessOutputParameters(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 925
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 518
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 279
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 326
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 94

 

问题分析

查看异常日志:

1)从最后一行看, 根据方法 Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync 可以得出,代码已经进入Function平台级别。可以初步排除是自己写的代码错误。

2)在逐行上看,发现 C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.SendGrid\Client\SendGridClient.cs : 23 中,调用了 Client.SendGridClient.SendMessageAsync(SendGridMessage msg,CancellationToken cancellationToken) 并抛出异常。

因为Azure Funciton的Source Code已经开源,所以可以在Github中查看到 WebJobs.Extensions.SendGrid\Client\SendGridClient.cs 的源代码。

注意: 代码中直接调用SendGird SDK中的Client对象,发送邮件 SendEmailAsync。

 

3) 继续向上查看,发现进入 SendGrid SendGrid.SendGridClient.MakeRequest(HttpRequestMessage request,CancellationToken cancellationToken) 中。同第二步一样,继续在Github中搜索SendGrid的源码,查看MakeRequest方法中进行了那些操作。

 

 

注意:方法MakeRequest中也没有多余配置,只是更深一步传递 Request请求。然后对获取的结果进行异常处理或成功还回,外加设置响应编码为UTF8

 

4) 继续向上查看,进入 SendGrid.Helpers.Reliability.RetryDelegatingHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken) 中。在此,发现可以设置以下四种条件的Retry次数。

private static readonly List<HttpStatusCode> RetriableServerErrorStatusCodes =
            new List<HttpStatusCode>()
            {
                HttpStatusCode.InternalServerError,
                HttpStatusCode.BadGateway,
                HttpStatusCode.ServiceUnavailable,
                HttpStatusCode.GatewayTimeout,
            };

RetryDeletegatingHandler.SendAsync 代码:

注意:如果设置了MaximumNumberOfRetries值,则当发送请求到SendGrid不成功时,会自动重试所设置的次数。重试次数必须在0 ~ 5 次之间。

if (maximumNumberOfRetries < 0)
            {
                throw new ArgumentOutOfRangeException(nameof(maximumNumberOfRetries), "maximumNumberOfRetries must be greater than 0");
            }
            if (maximumNumberOfRetries > 5)
            {
                throw new ArgumentOutOfRangeException(nameof(maximumNumberOfRetries), "The maximum number of retries allowed is 5");
            }

 

5) 继续向上查看,发现后期的方法全是 System.Net.Http 类中对HTTP请求的处理。至此,代码查看到达源端。

 

6)回看异常消息 System.Threading.Tasks.TaskCanceledException : The operation was canceled.  这是因为参数 cancellationToken 的原因。因为请求为异步 Task操作。当在请求长时间没有处理完成并且cancellationToken中设置的取消时间已到,当前任务会被取消并抛出以上消息:The operation was canceled.

 

为什么需要取消令牌(CancellationToken) ?

因为Task没有方法支持在外部取消Task,只能通过一个公共变量存放线程的取消状态,在线程内部通过变量判断线程是否被取消,当CancellationToken是取消状态,Task内部未启动的任务不会启动新线程。

(Source: https://www.cnblogs.com/fanfan-90/p/12660996.html)

 

通过查看源码,我们找到了为什么出现TaskCanceledException的异常,但是为什么 Function App 没有设置SendGrid的重试参数呢? 继续查看源码,发现,Function App在初始化SendGrid Client对象时,并没有做任何的配置修改。所以Retry默认保持为0.

/// <summary>
/// Gets the maximum number of retries to execute against when sending an HTTP Request before throwing an exception. 
/// Defaults to 0 (no retries, you must explicitly enable).
/// </summary>
public int MaximumNumberOfRetries { get; }

Azure Function 的 SendGridOptions.cs 源文件中,并没有任何MaximumNumberOfRetries设置。

// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.
using SendGrid.Helpers.Mail;
namespace Microsoft.Azure.WebJobs.Extensions.SendGrid
{
    /// <summary>
    /// Defines the configuration options for the SendGrid binding.
    /// </summary>
    public class SendGridOptions
    {
        /// <summary>
        /// Gets or sets the SendGrid ApiKey. If not explicitly set, the value will be defaulted
        /// to the value specified via the 'AzureWebJobsSendGridApiKey' app setting or the
        /// 'AzureWebJobsSendGridApiKey' environment variable.
        /// </summary>
        public string ApiKey { get; set; }
        /// <summary>
        /// Gets or sets the default "to" address that will be used for messages.
        /// This value can be overridden by job functions.
        /// </summary>
        /// <remarks>
        /// An example of when it would be useful to provide a default value for 'to' 
        /// would be for emailing your own admin account to notify you when particular
        /// jobs are executed. In this case, job functions can specify minimal info in
        /// their bindings, for example just a Subject and Text body.
        /// </remarks>
        public EmailAddress ToAddress { get; set; }
        /// <summary>
        /// Gets or sets the default "from" address that will be used for messages.
        /// This value can be overridden by job functions.
        /// </summary>
        public EmailAddress FromAddress { get; set; }
    }
}

 

参考资料

Azure Functions SendGrid 绑定: https://docs.azure.cn/zh-cn/azure-functions/functions-bindings-sendgrid?tabs=csharp

WebJobs.Extensions.SendGrid: https://github.com/Azure/azure-webjobs-sdk-extensions/tree/dev/src/WebJobs.Extensions.SendGrid

SendGrid: https://github.com/sendgrid/sendgrid-csharp/tree/main/src/SendGrid

多线程笔记-CancellationToken(取消令牌):https://www.cnblogs.com/fanfan-90/p/12660996.html

相关文章
|
1月前
【Azure Logic App】使用Event Hub 连接器配置 Active Directory OAuth 认证无法成功连接到中国区Event Hub的解决之法
An exception occurred while retrieving properties for Event Hub: logicapp. Error Message: 'ClientSecretCredential authentication failed: AADSTS90002: Tenant 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' not found. Check to make sure you have the correct tenant ID and are signing into the correct cloud. Che
|
1月前
|
安全
【Azure App Service】App service无法使用的情况分析
App Service集成子网后,如果子网网段中的剩余IP地址非常少的情况下,会在App Service实例升级时( 先加入新实例,然后在移除老实例 )。新加入的实例不能被分配到正确的内网IP地址,无法成功的访问内网资源。 解决方法就是为App Service增加子网地址, 最少需要/26 子网网段地址。
|
1月前
仿SOUL社交友附近人婚恋约仿陌陌APP网站源码
仿SOUL社交友附近人婚恋约仿陌陌APP网站源码
40 0
仿SOUL社交友附近人婚恋约仿陌陌APP网站源码
|
2月前
|
存储 BI Android开发
全开源仿第八区H5APP封装打包分发系统源码
全开源仿第八区H5APP封装打包分发系统源码
108 4
|
2月前
|
C++
【Azure Logic App】使用Event Hub 连接器配置 Active Directory OAuth 认证无法成功连接到中国区Event Hub
【Azure Logic App】使用Event Hub 连接器配置 Active Directory OAuth 认证无法成功连接到中国区Event Hub
|
2月前
【Azure Logic App】在逻辑应用中开启或关闭一个工作流是否会对其它工作流产生影响呢?
【Azure Logic App】在逻辑应用中开启或关闭一个工作流是否会对其它工作流产生影响呢?
|
2月前
|
存储 Linux 开发工具
【Azure App Service】本地Git部署Python Flask应用上云(Azure App Service For Linux)关键错误
【Azure App Service】本地Git部署Python Flask应用上云(Azure App Service For Linux)关键错误
|
2月前
|
存储 SQL JSON
【Azure Logic App】微软云逻辑应用连接到数据库,执行存储过程并转换执行结果为JSON数据
【Azure Logic App】微软云逻辑应用连接到数据库,执行存储过程并转换执行结果为JSON数据
【Azure Logic App】微软云逻辑应用连接到数据库,执行存储过程并转换执行结果为JSON数据
|
2月前
|
安全 网络安全 Windows
【Azure App Service】遇见az命令访问HTTPS App Service 时遇见SSL证书问题,暂时跳过证书检查的办法
【Azure App Service】遇见az命令访问HTTPS App Service 时遇见SSL证书问题,暂时跳过证书检查的办法
【Azure App Service】遇见az命令访问HTTPS App Service 时遇见SSL证书问题,暂时跳过证书检查的办法
|
2月前
|
安全 前端开发 网络安全
【Azure App Service】访问App Service应用报错 SSL: WRONG_VERSION_NUMBER
【Azure App Service】访问App Service应用报错 SSL: WRONG_VERSION_NUMBER
100 0

热门文章

最新文章