【Azure 微服务】Service Fabric 部署时遇见了VMExtensionProvisioningError错误: Multiple VM extensions failed to be provisioned on the VM

本文涉及的产品
日志服务 SLS,月写入数据量 50GB 1个月
简介: 【Azure 微服务】Service Fabric 部署时遇见了VMExtensionProvisioningError错误: Multiple VM extensions failed to be provisioned on the VM

问题描述

Deployment  Azure Service Fabric 时,遇见了VMExtensionProvisioningError, 全文如下:

Deployment Name: 385A084F35B2BC55-xxxx
Operation Id: 096CED
Operation: Create
Correlation Id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Status: Failed
Error:
Code: VMExtensionProvisioningError
Message: Multiple VM extensions failed to be provisioned on the VM. 
Please see the VM extension instance view for other failures. The first extension failed due to the error: 
VM Extension 'RunPowerShell_Frontdoor' is marked as failed since it depends upon the VM Extension 'ServiceFabricNodeVmExt_vmNodeType1Name' which has failed.
Help Link: https://aka.ms/ev2/errors/troubleshooting

 

问题解答

第一步:收集日志

需要远程连接(RDP)到SF的节点中, 在目录 C:\WindowsAzure\Logs\Plugins 中收集VM Extension的安装日志

 

第二步:分析日志

在 Microsoft.Azure.ServiceFabric.ServiceFabricNode plugin 日志中,发现是由于在获取证书失败而出现的错误。

[09/09/2021 12:32:01.86] Executing: C:\Packages\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11\ServiceFabricExtensionHandler.exe enable
[09/09/2021 12:32:03.62] Execution Complete.
######
Execution Output:
1> 2021-09-09T12:32:01.9973303Z: Information: Starting...
1> 2021-09-09T12:32:01.9973303Z: Information:   MachineName:       Frontdoor000000
1> 2021-09-09T12:32:01.9973303Z: Information:   CurrentDirectory:  C:\Packages\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11
1> 2021-09-09T12:32:01.9973303Z: Information:   UserName:          SYSTEM
1> 2021-09-09T12:32:01.9973303Z: Information: Created mutex; this is the only instance of this process running
1> 2021-09-09T12:32:02.3723445Z: Information: HandlerManifest Version: 1
1> 2021-09-09T12:32:02.4348520Z: Information: [HandlerEnvironment]
1> 2021-09-09T12:32:02.4348520Z: Information:   LogFolder:     C:\WindowsAzure\Logs\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11
1> 2021-09-09T12:32:02.4348520Z: Information:   ConfigFolder:  C:\Packages\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11\RuntimeSettings
1> 2021-09-09T12:32:02.4348520Z: Information:   StatusFolder:  C:\Packages\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11\Status
1> 2021-09-09T12:32:02.4348520Z: Information:   HeartbeatFile: C:\Packages\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11\Status\HeartBeat.Json
1> 2021-09-09T12:32:02.4348520Z: Information:   DeploymentId:  xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
1> 2021-09-09T12:32:02.4348520Z: Information:   RoleName:      _Frontdoor_0
1> 2021-09-09T12:32:02.4348520Z: Information:   Instance:      _Frontdoor_0
1> 2021-09-09T12:32:02.5442327Z: Information: Opening config file: C:\Packages\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11\RuntimeSettings\1.settings
1> 2021-09-09T12:32:02.5598642Z: Information: Config specified client cert FindBySubjectName 'xxxxx.cluster.prod.ngpproxy.microsoftonline.cn' (store name = My)
1> 2021-09-09T12:32:02.5754780Z: Error: ERROR: Microsoft.Azure.ServiceFabric.Extension.Core.AgentException: Cannot find certificate with 'FindBySubjectName' 'xxxxx.cluster.prod.ngpproxy.microsoftonline.cn' in store 'My'    at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadClientCertificate(String findValue, String certStoreName, X509FindType findBy)    at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadCertificateWrapper(CertificateSettings certificateSetting)
1> 2021-09-09T12:32:02.5754780Z: Warning: Heartbeat: Not Ready: Cannot find certificate with 'FindBySubjectName' 'xxxxx.cluster.prod.ngpproxy.microsoftonline.cn' in store 'My'
1> 2021-09-09T12:32:02.5754780Z: Error: Heartbeat file updated, with error msg
1> 2021-09-09T12:32:02.5911044Z: Error: ERROR: Microsoft.Azure.ServiceFabric.Extension.Core.AgentException: Cannot find certificate with 'FindBySubjectName' 'xxxxx.cluster.prod.ngpproxy.microsoftonline.cn' in store 'My'    at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadClientCertificate(String findValue, String certStoreName, X509FindType findBy)    at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadCertificateWrapper(CertificateSettings certificateSetting)    at Microsoft.Azure.ServiceFabric.Extension.Core.Common.PublicSettingsExtensions.AllowAccessToCerts(PublicSettings publicSettings, ICertificateUtility certificateUtility)    at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.ValidateDeployment(HandlerSettings settings)    at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.InstallService()    at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.InstallServiceWithRetry()    at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.Enable()    at Microsoft.Azure.ServiceFabric.Extension.Handler.Program.Main(String[] args) in X:\bt\1162492\repo\src\HandlerExe\Program.cs:line 52
Execution Error:
Microsoft.Azure.ServiceFabric.Extension.Core.AgentException: Cannot find certificate with 'FindBySubjectName' 'xxxxx.cluster.prod.ngpproxy.microsoftonline.cn' in store 'My'
   at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadClientCertificate(String findValue, String certStoreName, X509FindType findBy)
   at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadCertificateWrapper(CertificateSettings certificateSetting)
   at Microsoft.Azure.ServiceFabric.Extension.Core.Common.PublicSettingsExtensions.AllowAccessToCerts(PublicSettings publicSettings, ICertificateUtility certificateUtility)
   at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.ValidateDeployment(HandlerSettings settings)
   at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.InstallService()
   at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.InstallServiceWithRetry()
   at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.Enable()
   at Microsoft.Azure.ServiceFabric.Extension.Handler.Program.Main(String[] args) in X:\bt\1162492\repo\src\HandlerExe\Program.cs:line 52

最最关键的信息就是 Microsoft.Azure.ServiceFabric.Extension.Core.AgentException: Cannot find certificate with 'FindBySubjectName' 'xxxxx.cluster.prod.ngpproxy.microsoftonline.cn' in store 'My' 。

 

第三步:继续查找不能找到证书的问题

在获取证书的日志中,发现SF是从KV(Key Vault: 密钥保管库)中获取的证书,而在第5步获取auth challenge时,从18:59分02秒 到 19:04分45秒,耗时超过了5 分钟。

  1. 2021-09-09 18:59:02: <info> [CertificateManager] Beginning refresh for: https://xxxxx-chinanorth-kv.vault.azure.cn/secrets/cluster
  2. 2021-09-09 18:59:02: <info> [WindowsCertificateStore] attempting to open store 'LocalMachine\MY'
  3. 2021-09-09 18:59:02: <debug> [WindowsCertificateStore] opening the 'LocalMachine' store..
  4. 2021-09-09 18:59:02: <debug> [WindowsCertificateStore] store opened successfully.
  5. 2021-09-09 18:59:02: <info> [KeyVaultClient] Getting new auth challenge
  6. 2021-09-09 19:04:45: <error> [CertificateManager] Refreshing 'https://xxxxx-chinanorth-kv.vault.azure.cn/secrets/cluster' failed with KeyVault exception: KeyVaultHttpClient.GetChallenge. - http_exception: what = 'WinHttpSendRequest: 12030: The connection with the server was terminated abnormall' code = 'windows:12030'
  7. 2021-09-09 19:04:45: <info> [CertificateManager] Beginning refresh for:

由此可以推断,由于网络的延迟导致部署SF时,向Key Vault请求认证,由于请求超时而导致部署失败。

 

在多次重试重新部署SF,部署成功。

相关实践学习
日志服务之使用Nginx模式采集日志
本文介绍如何通过日志服务控制台创建Nginx模式的Logtail配置快速采集Nginx日志并进行多维度分析。
相关文章
|
28天前
|
Kubernetes 持续交付 Docker
利用 Docker 和 Kubernetes 实现微服务部署
【10月更文挑战第2天】利用 Docker 和 Kubernetes 实现微服务部署
|
3月前
|
SQL Kubernetes 监控
微服务从代码到k8s部署应有尽有系列(九、事务精讲)
微服务从代码到k8s部署应有尽有系列(九、事务精讲)
微服务从代码到k8s部署应有尽有系列(九、事务精讲)
|
2月前
|
运维 Kubernetes Cloud Native
云原生时代下,如何高效构建与部署微服务
【9月更文挑战第8天】随着云计算技术的飞速发展,云原生已成为现代软件架构的重要趋势。本文将深入浅出地介绍云原生概念、微服务架构的优势以及如何在云平台上高效构建和部署微服务。我们将通过实际的代码示例,展示在Kubernetes集群上部署一个简单的微服务应用的过程,帮助读者理解云原生环境下的微服务开发和运维实践。
|
3月前
|
Kubernetes jenkins 持续交付
微服务从代码到k8s部署应有尽有系列(十四、部署环境搭建)
微服务从代码到k8s部署应有尽有系列(十四、部署环境搭建)
|
3月前
|
负载均衡 应用服务中间件 持续交付
微服务架构下的Web服务器部署
【8月更文第28天】随着互联网应用的不断发展,传统的单体应用架构逐渐显露出其局限性,特别是在可扩展性和维护性方面。为了解决这些问题,微服务架构应运而生。微服务架构通过将应用程序分解成一系列小型、独立的服务来提高系统的灵活性和可维护性。本文将探讨如何在微服务架构中有效部署和管理Web服务器实例,并提供一些实际的代码示例。
102 0
|
3月前
|
Kubernetes 监控 中间件
微服务从代码到k8s部署应有尽有系列全集
微服务从代码到k8s部署应有尽有系列全集
|
3月前
|
消息中间件 Kubernetes Kafka
微服务从代码到k8s部署应有尽有系列(八、各种队列)
微服务从代码到k8s部署应有尽有系列(八、各种队列)
|
2月前
|
Dubbo Java 应用服务中间件
微服务框架Dubbo环境部署实战
微服务框架Dubbo环境部署的实战指南,涵盖了Dubbo的概述、服务部署、以及Dubbo web管理页面的部署,旨在指导读者如何搭建和使用Dubbo框架。
203 17
微服务框架Dubbo环境部署实战
|
1月前
|
Kubernetes Docker 微服务
微服务实践k8s&dapr开发部署实验(1)服务调用(一)
微服务实践k8s&dapr开发部署实验(1)服务调用(一)
45 2
|
30天前
|
Kubernetes Docker 微服务
微服务实践k8s&dapr开发部署实验(1)服务调用(二)
微服务实践k8s&dapr开发部署实验(1)服务调用(二)
47 0

热门文章

最新文章