问题描述
在使用Azure Function App的 SendGrid Binging 功能，调用SendGrid服务器发送邮件功能时，遇到见间歇性，偶发性的异常。在重新运行SendGrid的Function，却又能恢复运行。
所以本文基于Azure Function使用SendGrid的异常错误日志，一步一步，分析源码中的方法内容。 然后调查为什么 Azure Function 没有自动重试(Retry)？
(如需要参考如何使用Azure Function SendGrid，参考：Azure Functions SendGrid 绑定)
错误消息：
System.Threading.Tasks.TaskCanceledException : The operation was canceled. at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async System.Net.Http.ConnectHelper.ConnectAsync(String host,Int32 port,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Threading.Tasks.ValueTask`1.get_Result() at async System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request,Boolean allowHttp2,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Threading.Tasks.ValueTask`1.get_Result() at async System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Threading.Tasks.ValueTask`1.get_Result() at async System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Threading.Tasks.ValueTask`1.get_Result() at async System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request,Boolean doRequestAuth,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async SendGrid.Helpers.Reliability.RetryDelegatingHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask,HttpRequestMessage request,CancellationTokenSource cts,Boolean disposeCts) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async SendGrid.SendGridClient.MakeRequest(HttpRequestMessage request,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async SendGrid.SendGridClient.RequestAsync(Method method,String requestBody,String queryParams,String urlPath,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async SendGrid.SendGridClient.SendEmailAsync(SendGridMessage msg,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Client.SendGridClient.SendMessageAsync(SendGridMessage msg,CancellationToken cancellationToken) at C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.SendGrid\Client\SendGridClient.cs : 23 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Extensions.Bindings.SendGridMessageAsyncCollector.FlushAsync(CancellationToken cancellationToken) at C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.SendGrid\Bindings\SendGridMessageAsyncCollector.cs : 56 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Bindings.AsyncCollectorValueProvider`2.SetValueAsync[TUser,TMessage](Object value,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Bindings\AsyncCollector\AsyncCollectorValueProvider.cs : 49 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Binder.Complete(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Bindings\Runtime\Binder.cs : 150 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Bindings.Runtime.RuntimeValueProvider.SetValueAsync(Object value,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Bindings\Runtime\RuntimeValueProvider.cs : 34 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ParameterHelper.ProcessOutputParameters(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 925 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 518 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 279 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 326 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 94
问题分析
查看异常日志：
1）从最后一行看， 根据方法 Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync 可以得出，代码已经进入Function平台级别。可以初步排除是自己写的代码错误。
2）在逐行上看，发现 C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.SendGrid\Client\SendGridClient.cs : 23 中，调用了 Client.SendGridClient.SendMessageAsync(SendGridMessage msg,CancellationToken cancellationToken) 并抛出异常。
因为Azure Funciton的Source Code已经开源，所以可以在Github中查看到 WebJobs.Extensions.SendGrid\Client\SendGridClient.cs 的源代码。
注意： 代码中直接调用SendGird SDK中的Client对象，发送邮件 SendEmailAsync。
3) 继续向上查看，发现进入 SendGrid 的 SendGrid.SendGridClient.MakeRequest(HttpRequestMessage request,CancellationToken cancellationToken) 中。同第二步一样，继续在Github中搜索SendGrid的源码，查看MakeRequest方法中进行了那些操作。
注意：方法MakeRequest中也没有多余配置，只是更深一步传递 Request请求。然后对获取的结果进行异常处理或成功还回，外加设置响应编码为UTF8
4) 继续向上查看，进入 SendGrid.Helpers.Reliability.RetryDelegatingHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken) 中。在此，发现可以设置以下四种条件的Retry次数。
private static readonly List<HttpStatusCode> RetriableServerErrorStatusCodes = new List<HttpStatusCode>() { HttpStatusCode.InternalServerError, HttpStatusCode.BadGateway, HttpStatusCode.ServiceUnavailable, HttpStatusCode.GatewayTimeout, };
RetryDeletegatingHandler.SendAsync 代码：
注意：如果设置了MaximumNumberOfRetries值，则当发送请求到SendGrid不成功时，会自动重试所设置的次数。重试次数必须在0 ~ 5 次之间。
if (maximumNumberOfRetries < 0) { throw new ArgumentOutOfRangeException(nameof(maximumNumberOfRetries), "maximumNumberOfRetries must be greater than 0"); } if (maximumNumberOfRetries > 5) { throw new ArgumentOutOfRangeException(nameof(maximumNumberOfRetries), "The maximum number of retries allowed is 5"); }
5) 继续向上查看，发现后期的方法全是 System.Net.Http 类中对HTTP请求的处理。至此，代码查看到达源端。
6）回看异常消息 System.Threading.Tasks.TaskCanceledException : The operation was canceled. 这是因为参数 cancellationToken 的原因。因为请求为异步 Task操作。当在请求长时间没有处理完成并且cancellationToken中设置的取消时间已到，当前任务会被取消并抛出以上消息：The operation was canceled.
为什么需要取消令牌(CancellationToken) ？
因为Task没有方法支持在外部取消Task，只能通过一个公共变量存放线程的取消状态，在线程内部通过变量判断线程是否被取消，当CancellationToken是取消状态，Task内部未启动的任务不会启动新线程。
通过查看源码，我们找到了为什么出现TaskCanceledException的异常，但是为什么 Function App 没有设置SendGrid的重试参数呢？ 继续查看源码，发现，Function App在初始化SendGrid Client对象时，并没有做任何的配置修改。所以Retry默认保持为0.
/// <summary> /// Gets the maximum number of retries to execute against when sending an HTTP Request before throwing an exception. /// Defaults to 0 (no retries, you must explicitly enable). /// </summary> public int MaximumNumberOfRetries { get; }
Azure Function 的 SendGridOptions.cs 源文件中，并没有任何MaximumNumberOfRetries设置。
// Copyright (c) .NET Foundation. All rights reserved. // Licensed under the MIT License. See License.txt in the project root for license information. using SendGrid.Helpers.Mail; namespace Microsoft.Azure.WebJobs.Extensions.SendGrid { /// <summary> /// Defines the configuration options for the SendGrid binding. /// </summary> public class SendGridOptions { /// <summary> /// Gets or sets the SendGrid ApiKey. If not explicitly set, the value will be defaulted /// to the value specified via the 'AzureWebJobsSendGridApiKey' app setting or the /// 'AzureWebJobsSendGridApiKey' environment variable. /// </summary> public string ApiKey { get; set; } /// <summary> /// Gets or sets the default "to" address that will be used for messages. /// This value can be overridden by job functions. /// </summary> /// <remarks> /// An example of when it would be useful to provide a default value for 'to' /// would be for emailing your own admin account to notify you when particular /// jobs are executed. In this case, job functions can specify minimal info in /// their bindings, for example just a Subject and Text body. /// </remarks> public EmailAddress ToAddress { get; set; } /// <summary> /// Gets or sets the default "from" address that will be used for messages. /// This value can be overridden by job functions. /// </summary> public EmailAddress FromAddress { get; set; } } }
参考资料
Azure Functions SendGrid 绑定: https://docs.azure.cn/zh-cn/azure-functions/functions-bindings-sendgrid?tabs=csharp
WebJobs.Extensions.SendGrid: https://github.com/Azure/azure-webjobs-sdk-extensions/tree/dev/src/WebJobs.Extensions.SendGrid
SendGrid: https://github.com/sendgrid/sendgrid-csharp/tree/main/src/SendGrid
多线程笔记-CancellationToken（取消令牌）：https://www.cnblogs.com/fanfan-90/p/12660996.html
