前言
k8s分二进制安装与kubeadm安装，本次我刚好有两套部署好的环境，就都试一下减容与缩容
二进制
[root@cm1 pv]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME cm1 Ready <none> 3d20h v1.25.0 10.10.21.197 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21 cm2 Ready <none> 3d20h v1.25.0 10.10.21.198 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21 cm3 Ready <none> 3d20h v1.25.0 10.10.21.199 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21 cn1 Ready <none> 3d20h v1.25.0 10.10.21.200 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21 cn2 Ready <none> 3d20h v1.25.0 10.10.21.201 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21 cn3 Ready <none> 3d20h v1.25.0 10.10.21.202 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21 cn4 Ready <none> 7m v1.25.0 10.10.21.205 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21
kubeadm
root@master1:~/rbd_yaml# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master1.hu.org Ready control-plane 37d v1.25.2 10.10.21.170 <none> Ubuntu 20.04.5 LTS 5.4.0-131-generic docker://20.10.12 master2.hu.org Ready control-plane 36d v1.25.2 10.10.21.172 <none> Ubuntu 20.04.5 LTS 5.4.0-90-generic docker://20.10.12 master3.hu.org Ready control-plane 36d v1.25.2 10.10.21.175 <none> Ubuntu 20.04.5 LTS 5.4.0-90-generic docker://20.10.12 node1.hu.org Ready <none> 37d v1.25.2 10.10.21.171 <none> Ubuntu 20.04.5 LTS 5.4.0-90-generic docker://20.10.12 node2.hu.org Ready <none> 37d v1.25.2 10.10.21.173 <none> Ubuntu 20.04.5 LTS 5.4.0-90-generic docker://20.10.12 node3.hu.org Ready <none> 37d v1.25.2 10.10.21.176 <none> Ubuntu 20.04.5 LTS 5.4.0-90-generic docker://20.10.12
缩容
root@master1:~/rbd_yaml# kubectl get node NAME STATUS ROLES AGE VERSION master1.hu.org Ready control-plane 37d v1.25.2 master2.hu.org Ready control-plane 36d v1.25.2 master3.hu.org Ready control-plane 36d v1.25.2 node1.hu.org Ready <none> 37d v1.25.2 node2.hu.org Ready <none> 37d v1.25.2 node3.hu.org Ready <none> 37d v1.25.2 root@master1:~/rbd_yaml# kubectl cordon node3.hu.org # 设置节点不可调度 node/node3.hu.org cordoned root@master1:~/rbd_yaml# kubectl drain node3.hu.org --delete-emptydir-data --ignore-daemonsets # 驱逐pod node/node3.hu.org already cordoned Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-ff5k4, kube-system/kube-proxy-5pnh5, monitor/node-exporter-65pmz evicting pod monitor/blackbox-exporter-59b7cc76b-br7p9 evicting pod kubegems/kubegems-argo-cd-app-controller-777d8fffb-4l6nr evicting pod default/nginx-6768c68f7b-dnt7z evicting pod default/nginx2-b648d744f-kp925 evicting pod kube-system/coredns-c676cc86f-7l2c8 evicting pod kubegems-local/kubegems-local-kubectl-bf9f98658-7t4jv evicting pod kubegems/kubegems-mysql-0 evicting pod kubegems/kubegems-dashboard-78bd4449c4-ghn2g evicting pod kubegems/kubegems-gitea-0 evicting pod kubegems/kubegems-redis-master-0 evicting pod kubernetes-dashboard/kubernetes-dashboard-566f484578-gmnnt pod/kubegems-dashboard-78bd4449c4-ghn2g evicted pod/kubernetes-dashboard-566f484578-gmnnt evicted I1118 02:41:07.415515 137843 request.go:682] Waited for 1.088277469s due to client-side throttling, not priority and fairness, request: GET:https://10.10.21.178:6443/api/v1/namespaces/kubegems/pods/kubegems-argo-cd-app-controller-777d8fffb-4l6nr pod/nginx-6768c68f7b-dnt7z evicted pod/blackbox-exporter-59b7cc76b-br7p9 evicted pod/kubegems-local-kubectl-bf9f98658-7t4jv evicted pod/nginx2-b648d744f-kp925 evicted pod/kubegems-redis-master-0 evicted pod/kubegems-argo-cd-app-controller-777d8fffb-4l6nr evicted pod/kubegems-gitea-0 evicted pod/kubegems-mysql-0 evicted pod/coredns-c676cc86f-7l2c8 evicted node/node3.hu.org drained root@master1:~/rbd_yaml# kubectl get node NAME STATUS ROLES AGE VERSION master1.hu.org Ready control-plane 37d v1.25.2 master2.hu.org Ready control-plane 36d v1.25.2 master3.hu.org Ready control-plane 36d v1.25.2 node1.hu.org Ready <none> 37d v1.25.2 node2.hu.org Ready <none> 37d v1.25.2 node3.hu.org Ready,SchedulingDisabled <none> 37d v1.25.2 root@master1:~/rbd_yaml# kubectl delete node node3.hu.org node "node3.hu.org" deleted root@master1:~/rbd_yaml# kubectl get node NAME STATUS ROLES AGE VERSION master1.hu.org Ready control-plane 37d v1.25.2 master2.hu.org Ready control-plane 36d v1.25.2 master3.hu.org Ready control-plane 36d v1.25.2 node1.hu.org Ready <none> 37d v1.25.2 node2.hu.org Ready <none> 37d v1.25.2
扩容
以下操作默认已经装好了容器引擎，完成了安装前必要条件
二进制扩容
拷贝证书到新节点上
[root@cn4 kubernetes]# pwd /etc/kubernetes [root@cn4 kubernetes]# scp -r cn3:/etc/kubernetes/* . root@cn3's password: bootstrap-kubelet.kubeconfig 100% 2232 3.5MB/s 00:00 kubelet-conf.yml 100% 1628 4.9MB/s 00:00 kubelet.kubeconfig 100% 2302 8.4MB/s 00:00 kube-proxy.kubeconfig 100% 6381 18.6MB/s 00:00 kube-proxy.yaml 100% 825 3.6MB/s 00:00 ca.pem 100% 1363 5.2MB/s 00:00 ca-key.pem 100% 1675 1.1MB/s 00:00 front-proxy-ca.pem
启动kubelet和kube-proxy
[root@cn4 kubernetes]# systemctl start kubelet kube-proxy.service [root@cn4 kubernetes]# systemctl status kubelet kube-proxy.service ● kubelet.service - Kubernetes Kubelet Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2022-11-18 11:03:16 CST; 9s ago Docs: https://github.com/kubernetes/kubernetes Main PID: 70687 (kubelet) Tasks: 14 Memory: 31.4M CGroup: /system.slice/kubelet.service └─70687 /usr/local/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/kub... Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956399 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-local-net-dir\" (UniqueNa... Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956482 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2glx\" (Uniqu... Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956497 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"k... Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956513 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubern... Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956552 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bpffs\" (UniqueName: \"kuberne... Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956572 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-bin-dir\" (UniqueName: \"k... Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956586 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-net-dir\" (UniqueName: \"k... Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956626 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-log-dir\" (UniqueName: \"k... Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956642 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"policysync\" (UniqueName: \"ku... Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956650 70687 reconciler.go:169] "Reconciler: start to sync state" ● kube-proxy.service - Kubernetes Kube Proxy Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2022-11-18 11:03:16 CST; 9s ago Docs: https://github.com/kubernetes/kubernetes Main PID: 70688 (kube-proxy) Tasks: 5 Memory: 11.1M CGroup: /system.slice/kube-proxy.service └─70688 /usr/local/bin/kube-proxy --config=/etc/kubernetes/kube-proxy.yaml --v=2 Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884482 70688 service.go:440] "Adding new service port" portName="kube-system/calico-typha:calico-typha" servicePort="1...67:5473/TCP" Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884491 70688 service.go:440] "Adding new service port" portName="kube-system/kube-dns:dns" servicePort="10.96.0.10:53/UDP" Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884498 70688 service.go:440] "Adding new service port" portName="kube-system/kube-dns:dns-tcp" servicePort="10.96.0.10:53/TCP" Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884504 70688 service.go:440] "Adding new service port" portName="kube-system/kube-dns:metrics" servicePort="10.96.0.10:9153/TCP" Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884568 70688 service.go:440] "Adding new service port" portName="kube-system/metrics-server:https" servicePort="10.97....193:443/TCP" Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884579 70688 service.go:440] "Adding new service port" portName="ingress-nginx/ingress-nginx-controller:http" serviceP....189:80/TCP" Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884587 70688 service.go:440] "Adding new service port" portName="ingress-nginx/ingress-nginx-controller:https" service...189:443/TCP" Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884595 70688 service.go:440] "Adding new service port" portName="kube-system/default-http-backend" servicePort="10.107....129:80/TCP" Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884605 70688 service.go:440] "Adding new service port" portName="default/kubernetes:https" servicePort="10.96.0.1:443/TCP" Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884855 70688 proxier.go:1030] "Stale service" protocol="udp" servicePortName="kube-system/kube-dns:dns" clusterIP="10.96.0.10" Hint: Some lines were ellipsized, use -l to show in full. # 没报错的话大概率成了，我这边因为证书之前有记录过这个节点，开启服务就直接扩进去了 # 如果没有记录的话kubelt会提示node名字找不到，这个时候需要到master节点操作 kubectl get csr kubectl certificate approve csr-vx866 # csr是上面get出来的
扩容完成检查一下
[root@cm1 pv]# kubectl get node,cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS ROLES AGE VERSION node/cm1 Ready <none> 3d21h v1.25.0 node/cm2 Ready <none> 3d21h v1.25.0 node/cm3 Ready <none> 3d21h v1.25.0 node/cn1 Ready <none> 3d21h v1.25.0 node/cn2 Ready <none> 3d21h v1.25.0 node/cn3 Ready <none> 3d21h v1.25.0 node/cn4 Ready <none> 7m50s v1.25.0 NAME STATUS MESSAGE ERROR componentstatus/scheduler Healthy ok componentstatus/controller-manager Healthy ok componentstatus/etcd-2 Healthy {"health":"true","reason":""} componentstatus/etcd-1 Healthy {"health":"true","reason":""} componentstatus/etcd-0 Healthy {"health":"true","reason":""}
kubeadm扩容
master节点生成token
root @master1:~/rbd_yaml# kubeadm token create --print-join-command kubeadm join 10.10.21.178:6443 --token uo89v1.99wf83u43ntilt41 --discovery-token-ca-cert-hash sha256:44aba1ef82b6b34c40fe748a9c2cd321be91aa3c22dd23e706001b65affb9dc9
新节点扩进来
登录到需要扩容的节点上
root@node3:~# kubeadm join 10.10.21.178:6443 --token uo89v1.99wf83u43ntilt41 --discovery-token-ca-cert-hash sha256:44aba1ef82b6b34c40fe748a9c2cd321be91aa3c22dd23e706001b65affb9dc9 Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/cri-dockerd.sock To see the stack trace of this error execute with --v=5 or higher #直接执行master上生成的指令报错
这是因为我容器引擎是docker，我们这边的cri接口不上containerd，所以真实命令应该为
root@node3:~# kubeadm join 10.10.21.178:6443 --token uo89v1.99wf83u43ntilt41 --discovery-token-ca-cert-hash sha256:44aba1ef82b6b34c40fe748a9c2cd321be91aa3c22dd23e706001b65affb9dc9 --cri-socket unix:///run/cri-dockerd.sock [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' W1118 02:53:23.454899 2669242 cluster.go:94] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta3", Kind:"ClusterConfiguration"}: strict decoding error: unknown field "apiServer.certSANS" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
扩容成功检查一下
root@master1:~/rbd_yaml# kubectl get node,cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS ROLES AGE VERSION node/master1.hu.org Ready control-plane 37d v1.25.2 node/master2.hu.org Ready control-plane 36d v1.25.2 node/master3.hu.org Ready control-plane 36d v1.25.2 node/node1.hu.org Ready <none> 37d v1.25.2 node/node2.hu.org Ready <none> 37d v1.25.2 node/node3.hu.org Ready <none> 18m v1.25.2 NAME STATUS MESSAGE ERROR componentstatus/scheduler Healthy ok componentstatus/controller-manager Healthy ok componentstatus/etcd-2 Healthy {"health":"true"} componentstatus/etcd-0 Healthy {"health":"true"} componentstatus/etcd-1 Healthy {"health":"true"}
小提示
如果遇到如以下的报错，多半是node节点之前没有清除好
[preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists [ERROR Port-10250]: Port 10250 is in use [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
这时候需要将/etc/kubernetes/下清除掉，然后再关闭kubelet再执行扩容命令即可
