一点题外话：kubernetes官方3月25号发布1.14，本文28号完成。1.14升级安装中文指南，目前全网大概最新吧，支持请赏个赞。
升级准备
本次升级主要参考官方Upgrading kubeadm clusters from v1.13 to v1.14
升级之前注意事项(翻译自官方文档)：
- 1.13.0以上，使用kubeadm部署的kubernetes集群
- Swap分区需要disabled
- 集群的控制平面个数和etcd pods需要是确定的。
- 认真阅读release notes。
- 备份。主要是备份一些重要的组件，比如database等。虽然升级不会调整业务负载，仅仅调整kubernetes，但是备份总是没错的。
- 所有的容器都会被重启，因为hash值会变化。
- 只能够进行小版本的升级，并且升级过程不能够跳级，比如从1.y到1.y+1,而不能够从1.y到1.y+2
按照要求查看kubernetes 1.14 更改说明重点阅读Urgent Upgrade Notes，结合自己业务，并没有发现特别重大的变动，可以放心升级。
kubernetes集群是按照kubernetes 1.13 全新安装指南搭建，如下:
[hall@192-168-10-21 ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION 192-168-10-14 Ready master 36h v1.13.0 192-168-10-18 Ready <none> 103d v1.13.0 192-168-10-21 Ready master 104d v1.13.0
业务数据备份，就不用介绍了。实际上安全起见最好先在测试集群上进行升级，通过后再考虑正式集群的升级。
升级过程主要变化的是kubernetes系统服务，重点是kubelet，所以将kubelet配置备份一下更为稳妥，方法如下：
1 查看kubelet服务配置:
[root@192-168-10-94 ~]# systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since 二 2019-03-19 18:38:46 CST; 1 weeks 1 days ago Docs: https://kubernetes.io/docs/ Main PID: 6033 (kubelet) Tasks: 17 Memory: 59.1M CGroup: /system.slice/kubelet.service └─6033 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=re...
2 查看服务的配置文件 10-kubeadm.conf :
[root@192-168-10-94 ~]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf # Note: This dropin only works with kubeadm and kubelet v1.11+ [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file. EnvironmentFile=-/etc/sysconfig/kubelet ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
3 备份涉及的配置文件
/etc/kubernetes/bootstrap-kubelet.conf (可能并不存在，没有也没有关系) /etc/kubernetes/kubelet.conf /var/lib/kubelet/kubeadm-flags.env /etc/sysconfig/kubelet
下面正式开始升级过程。
升级主控节点
升级之前，一定要确保具有多个控制节点，这样可以保障集群的可用。单一控制节点，升级万一挂了，怕是比较麻烦。添加控制节点的方法，参考上文的kubernetes 1.13 全新安装指南。
如果没有特殊说明本文除 kubectl 以外的命令，都是使用 root 账号执行。
1 先检查一下repo源中kubeadm是否更新到 1.14.0 的版本
yum list --showduplicates kubeadm --disableexcludes=kubernetes
我本地的源没有找到 1.14.0 。 使用下面命令清理，后再行检查可以得到 1.14.0
yum --disablerepo=\* --enablerepo=kubernetes clean all
2 再次查看kubeadm版本信息
[root@192-168-10-21 ~]# kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:35:32Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
3 安装kubeadm工具
yum install -y kubeadm-1.14.0-0 --disableexcludes=kubernetes
4 确认kubeadm版本升级完成
[root@192-168-10-21 ~]# kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:51:21Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
5 升级检查和方案
[root@192-168-10-21 ~]# kubeadm upgrade plan [preflight] Running pre-flight checks. [upgrade] Making sure the cluster is healthy: [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: v1.13.0 [upgrade/versions] kubeadm version: v1.14.0 Awesome, you're up-to-date! Enjoy! kubeadm upgrade apply v1.14.0
这里的提示信息和官方文档有出入，不过这是正常的信息。
6 升级kubeadm到1.14
kubeadm upgrade apply v1.14.0
这个执行过程，视集群情况，大概会执行几分钟，输出信息也比较多，大概如下:
[root@192-168-10-21 ~]# kubeadm upgrade apply v1.14.0 [preflight] Running pre-flight checks. [upgrade] Making sure the cluster is healthy: ..... [upgrade/staticpods] Component "kube-scheduler" upgraded successfully! [upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.14.0". Enjoy! [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
7 检查CNI情况，确定是否要升级
kubernetes架构，网络部分确定Container Network Interface接口，具体实现交由其它组件。我的集群使用flannel，检查一下：
kubectl get pods -n kube-system ... kubectl describe pod/kube-flannel-ds-amd64-5xxh7 -n kube-system ... Image: quay.io/coreos/flannel:v0.11.0-amd64
使用的是 0.11， 查看flannel主页得知已经是最新版，这一步不用处理。
8 升级kubectl和kubelet
yum install -y kubelet-1.14.0-0 kubectl-1.14.0-0 --disableexcludes=kubernetes 复制代码
9 重启kubelet
[root@192-168-10-21 ~]# systemctl restart kubelet Warning: kubelet.service changed on disk. Run 'systemctl daemon-reload' to reload units. [root@192-168-10-21 ~]# systemctl daemon-reload [root@192-168-10-21 ~]# systemctl restart kubelet
实际上，重启kubelet失败，报错：Failed to start ContainerManager failed to initialise top level QOS containers。 排查过程请见附1
10 检查升级结果
[hall@192-168-10-21 ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION 192-168-10-14 Ready master 38h v1.13.0 192-168-10-18 Ready <none> 103d v1.13.0 192-168-10-21 Ready master 104d v1.14.0
192-168-10-21的状态为 Ready ，版本也变为 1.14.0 ，主控节点升级成功。
其它控制节点升级
- 参考上文升级好kubeadm，kubectl和kubelet工具。
- 升级到1.14
主控节点已经执行了检查和升级，192-168-10-14只需要执行 kubeadm upgrade apply v1.14.0。
不幸的是，又遇到了一点状况 failed to get APIEndpoint information for this node，排查过程请见附2
- 重启kubelet
- 检查升级结果
[hall@192-168-10-21 ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION 192-168-10-14 Ready master 39h v1.14.0 192-168-10-18 Ready <none> 103d v1.13.0 192-168-10-21 Ready master 104d v1.14.0
业务节点升级
1 临时备份
因为集群就一个业务节点，为安全起见，先调整一个控制节点，用于临时支撑业务：
[tyhall51@192-168-10-21 ~]$ kubectl taint node 192-168-10-14 node-role.kubernetes.io/master- node/192-168-10-14 untainted
然后业务节点临时增加污点，防止升级期间调度：
[tyhall51@192-168-10-21 ~]$ kubectl drain 192-168-10-18 --ignore-daemonsets node/192-168-10-18 cordoned error: unable to drain node "192-168-10-18", aborting command... There are pending nodes to be drained: 192-168-10-18 error: cannot delete Pods with local storage (use --delete-local-data to override): kube-system/elasticsearch-logging-0, kube-system/elasticsearch-logging-1, kube-system/monitoring-influxdb-8b7d57f5c-2bhlw
2 安装kubeadm工具
3 升级kubedam到1.14
[root@192-168-10-18 ~]# kubeadm upgrade node config --kubelet-version v1.14.0 [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [upgrade] The configuration for this node was successfully updated! [upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
4 升级kubectl和kubelet
kubelet同样需要重启
5 还原临时备份
先取消业务节点污点
[tyhall51@192-168-10-21 ~]$ kubectl uncordon 192-168-10-18 node/192-168-10-18 uncordoned
然后还原master节点
[tyhall51@192-168-10-21 ~]$ kubectl taint node 192-168-10-14 node-role.kubernetes.io/master=:NoSchedule node/192-168-10-14 tainted
6 检查结果
[tyhall51@192-168-10-21 ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION 192-168-10-14 Ready master 40h v1.14.0 192-168-10-18 Ready <none> 103d v1.14.0 192-168-10-21 Ready master 104d v1.14.0
以上，完成了kubernetes从1.13到1.14的升级，整体上讲，升级过程比较轻松。总结一下升级过程:
- 详细阅读升级指南，完成重要业务信息备份，完成kubelet配置。
- 升级主要控制节点。
- 升级其它控制节点。
- 升级业务节点。
附
- kubelet重启失败
kubelet重启失败，systemctl status kubelet 中错误信息：
Failed to start ContainerManager failed to initialise top level QOS containers
参考https://github.com/kubernetes/kubernetes/issues/43704提示在kubelet启动时候增加 --cgroups-per-qos=false --enforce-node-allocatable="" 即可解决。之前备份kubelet的配置时候知道 /var/lib/kubelet/kubeadm-flags.env 中定义kubelet的启动参数，在其中加上，重启kubelet，恢复正常。
- kubeadm 升级失败
[root@192-168-10-14 ~]# kubeadm upgrade apply v1.14.0 [preflight] Running pre-flight checks. [upgrade] Making sure the cluster is healthy: [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [upgrade/config] FATAL: failed to getAPIEndpoint: failed to get APIEndpoint information for this node
根据提示，使用编辑 kubectl -n kube-system edit cm kubeadm-config -oyaml kubeadm-config, 调整apiEndpoints为:
apiEndpoints: 192-168-10-21: advertiseAddress: 192.168.10.21 bindPort: 6443 192-168-10-14: advertiseAddress: 192.168.10.14 bindPort: 6443
继续执行kubeadm upgrade apply v1.14.0，正常完成。
