问题描述
在Azure Kubernetes 服务中，创建一个Internal Load Balancer服务，使用以下yaml内容：
internallb.yaml
apiVersion: v1 kind: Service metadata: name: ilb-myapp annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "true" spec: type: LoadBalancer ports: - port: 80 selector: app: myapp
apply yaml
kubectl apply -f internallb.yaml
查看service状态，一直保持pending
kubectl get service ## 输出结果： NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ilb-myapp LoadBalancer 10.0.32.246 <pending> 80:30198/TCP 92m kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 13h
查看service日志
"error_description":"AADSTS7000222: The provided client secret keys for app '********-****-****-****-************' are expired kubectl describe service ilb-myapp ##输出结果： Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning UpdateLoadBalancerFailed 45m (x26 over 82m) service-controller (combined from similar events): Error updating load balancer with new hosts map[aks-vmss000002:{} aks-vmss000003:{} aks-vmss000006:{} aks-vmss000007:{}]: shouldUpdateLoadBalancer: failed to list managed load balancers: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 401, RawError: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to http://localhost:7788/*****: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys for app '********-****-****-****-************' are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds.\r\nTrace ID: fa5035d2-1a6e-4b5a-85d0-910862a12501\r\nCorrelation ID: a184c126-96cb-42a9-8c4b-92869210e295\r\nTimestamp: 2023-05-31 02:33:46Z","error_codes":[7000222],"timestamp":"2023-05-31 02:33:46Z","trace_id":"fa5035d2-1a6e-4b5a-85d0-910862a12501","correlation_id":"a184c126-96cb-42a9-8c4b-92869210e295","error_uri":"https://login.chinacloudapi.cn/error?code=7000222"} Normal EnsuringLoadBalancer 37s (x25 over 95m) service-controller Ensuring load balancer Warning ListLoadBalancers 37s (x152 over 91m) azure-cloud-provider (combined from similar events): Retriable: false, RetryAfter: 0s, HTTPStatusCode: 401, RawError: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to http://localhost:7788/*****: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys for app '********-****-****-****-************' are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds.\r\nTrace ID: 277e6d9c-d663-4415-bff9-ef6610dc4f01\r\nCorrelation ID: 383589a0-2dd7-4c8e-a1a3-23a48ae8bd6d\r\nTimestamp: 2023-05-31 03:19:01Z","error_codes":[7000222],"timestamp":"2023-05-31 03:19:01Z","trace_id":"277e6d9c-d663-4415-bff9-ef6610dc4f01","correlation_id":"383589a0-2dd7-4c8e-a1a3-23a48ae8bd6d","error_uri":"https://login.chinacloudapi.cn/error?code=7000222"}
问题解答
根据错误消息，已经得知是 app client secret 已经过期。那么要修复这个问题，就是更换新的 secret。操作步骤有两步：
第一步 ：根据错误消息中的Application ID，在Azure AD中查找到对应的注册应用
在注册应用中，重新生成新的Client Secret，复制保存新的Secret(用于第二步中)
第二步 : 使用 az aks update-credentials 命令，更新AKS集群中的密钥值(Secret Value)
az aks update-credentials --resource-group <resource group name> --name <AKS Cluster name> --reset-service-principal --service-principal <App Application ID> --client-secret <第一步中的Secret Value>
当命令执行完成后，在此检查 Service 状态，不在是Pending。而是分配了正确的IP地址
参考资料
故障排查：https://docs.azure.cn/zh-cn/aks/kubernetes-service-principal?tabs=azure-cli#troubleshoot
使用服务主体凭据更新 AKS 群集 ：https://docs.azure.cn/zh-cn/aks/update-credentials#update-aks-cluster-with-service-principal-credentials
