| IP | SERVICES |
| 192.168.72.55 | keepalived+nginx |
| 192.168.72.56 | keepalived+nginx |
| 192.168.72.57 | keepalived+nginx |
| 192.168.72.100 | VIP |
如果需要给kube-apiserver提供高可用，建议在部署kubernetes集群的时候就预先部署好keepalived+nginx，
- 否则的话，需要修改kube-apiserver的证书，将vip加入到证书内，
- 并且修改
kube-config相关的秘钥（kube-controller、kube-scheduler、kubelet、kube-proxy）都需要重新生成，将原有的kube-apiserver的认证地址修改为VIP - 重启
kube-apiserver、kube-controller、kube-scheduler、kubelet、kube-proxy服务，最终就可以实现高可用
- 所以，在kubernetes集群部署前先完成keepalived+nginx就会方便很多
为什么选择keepalived？
- 因为keepalived有后端服务
健康检测机制，检测到后端的nginx（nginx为了使用upstream模块实现负载均衡）服务有故障，就会将自己（keepalived）关闭，使VIP漂移到另外两个节点中的一个，用来保证和实现kube-apiserver服务的高可用 - 此次部署，是基于了我博客前面的
suse 12 二进制部署 Kubernetets 1.19.7系列，最后的分发脚本和启动服务里面的数组需要自行修改(修改成自己的ip或者主机名即可，主机名需要提前做好hosts解析)
编译部署nginx
下载nginx源码包
k8s-01:~ # cd /opt/k8s/packages/ k8s-01:/opt/k8s/packages # wget http://nginx.org/download/nginx-1.16.1.tar.gz k8s-01:/opt/k8s/packages # tar xf nginx-1.16.1.tar.gz
编译nginx
k8s-01:~ # cd /opt/k8s/packages/nginx-1.16.1/ k8s-01:/opt/k8s/packages/nginx-1.16.1 # ./configure --prefix=$(pwd)/nginx-prefix \ --with-stream \ --without-http \ --without-http_uwsgi_module && \ make && \ make install
配置nginx.conf
k8s-01:~ # cd /opt/k8s/conf/ k8s-01:/opt/k8s/conf # cat > kube-nginx.conf <<EOF worker_processes 1; events { worker_connections 1024; } stream { upstream backend { hash \$remote_addr consistent; server 192.168.72.55:6443 max_fails=3 fail_timeout=30s; server 192.168.72.56:6443 max_fails=3 fail_timeout=30s; server 192.168.72.57:6443 max_fails=3 fail_timeout=30s; } server { listen *:8443; proxy_connect_timeout 1s; proxy_pass backend; } } EOF
配置nginx为systemctl管理
k8s-01:~ # cd /opt/k8s/conf/ k8s-01:/opt/k8s/conf # cat > kube-nginx.service <<EOF [Unit] Description=kube-apiserver nginx proxy After=network.target After=network-online.target Wants=network-online.target [Service] Type=forking ExecStartPre=/opt/k8s/server/kube-nginx/sbin/nginx \ -c /opt/k8s/server/kube-nginx/conf/kube-nginx.conf \ -p /opt/k8s/server/kube-nginx -t ExecStart=/opt/k8s/server/kube-nginx/sbin/nginx \ -c /opt/k8s/server/kube-nginx/conf/kube-nginx.conf \ -p /opt/k8s/server/kube-nginx ExecReload=/opt/k8s/server/kube-nginx/sbin/nginx \ -c /opt/k8s/server/kube-nginx/conf/kube-nginx.conf \ -p /opt/k8s/server/kube-nginx -s reload PrivateTmp=true Restart=always RestartSec=5 StartLimitInterval=0 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
分发nginx二进制文件和配置文件
#!/usr/bin/env bash source /opt/k8s/bin/k8s-env.sh for host in ${MASTER_IPS[@]} do printf "\e[1;34m${host}\e[0m\n" ssh root@${host} "mkdir -p /opt/k8s/server/kube-nginx/{conf,logs,sbin}" scp /opt/k8s/packages/nginx-1.16.1/nginx-prefix/sbin/nginx ${host}:/opt/k8s/server/kube-nginx/sbin/ scp /opt/k8s/conf/kube-nginx.conf ${host}:/opt/k8s/server/kube-nginx/conf/ scp /opt/k8s/conf/kube-nginx.service ${host}:/etc/systemd/system/ done
启动kube-nginx服务
#!/usr/bin/env bash source /opt/k8s/bin/k8s-env.sh for host in ${MASTER_IPS[@]} do printf "\e[1;34m${host}\e[0m\n" ssh root@${host} "systemctl daemon-reload && \ systemctl enable kube-nginx --now && \ systemctl status kube-nginx | grep Active" done
编译部署keepalived
下载keepalived源码包
k8s-01:~ # cd /opt/k8s/packages/ k8s-01:/opt/k8s/packages # wget https://www.keepalived.org/software/keepalived-2.2.0.tar.gz k8s-01:/opt/k8s/packages # tar xf keepalived-2.2.0.tar.gz
编译keepalived
k8s-01:/opt/k8s/packages/keepalived-2.2.0 # ./configure --prefix=$(pwd)/keepalived-prefix && \ make && \ make install
配置keepalived.conf
k8s-01:~ # cd /opt/k8s/conf/ k8s-01:/opt/k8s/conf # cat > keepalived.conf.template <<EOF ! Configuration File for keepalived global_defs { } vrrp_script chk_nginx { script "/etc/keepalived/check_port.sh 8443" interval 2 weight -20 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 251 priority 100 advert_int 1 mcast_src_ip ##NODE_IP## nopreempt authentication { auth_type PASS auth_pass 11111111 } track_script { chk_nginx } virtual_ipaddress { 192.168.72.100 } } EOF
- 为了避免keepalived服务出现问题，修复后重启keepalived，出现IP漂移回来的情况，这里选择了3个都是BACKUP的模式，减少数据的丢失
创建健康检测脚本
k8s-01:~ # cd /opt/k8s/conf/ k8s-01:/opt/k8s/conf # cat > check_port.sh <<"EOF" CHK_PORT=$1 if [ -n "$CHK_PORT" ];then PORT_PROCESS=$(ss -lt|grep $CHK_PORT|wc -l) if [ $PORT_PROCESS -eq 0 ];then echo "Port $CHK_PORT Is Not Used,End." exit 1 fi else echo "Check Port Cant Be Empty!" fi EOF
配置keepalived为systemctl管理
k8s-01:~ # cd /opt/k8s/conf/ k8s-01:/opt/k8s/conf # cat > keepalived.service <<EOF [Unit] Description=LVS and VRRP High Availability Monitor After=syslog.target network-online.target [Service] Type=forking PIDFile=/var/run/keepalived.pid KillMode=process EnvironmentFile=-/etc/sysconfig/keepalived ExecStart=/usr/sbin/keepalived \$KEEPALIVED_OPTIONS ExecReload=/bin/kill -HUP \$MAINPID [Install] WantedBy=multi-user.target EOF
分发keepalived二进制文件和配置文件
#!/usr/bin/env bash source /opt/k8s/bin/k8s-env.sh for (( i=0; i < 3; i++ )) do sed -e "s/##NODE_IP##/${MASTER_IPS[i]}/" /opt/k8s/conf/keepalived.conf.template > \ /opt/k8s/conf/keepalived.conf-${MASTER_IPS[i]}.template done for host in ${MASTER_IPS[@]} do printf "\e[1;34m${host}\e[0m\n" ssh root@${host} "mkdir -p /etc/keepalived" scp /opt/k8s/packages/keepalived-2.2.0/keepalived-prefix/sbin/keepalived ${host}:/usr/sbin/ scp /opt/k8s/packages/keepalived-2.2.0/keepalived-prefix/etc/sysconfig/keepalived ${host}:/etc/sysconfig/ scp /opt/k8s/conf/keepalived.conf-${host}.template ${host}:/etc/keepalived/keepalived.conf scp /opt/k8s/conf/check_port.sh ${host}:/etc/keepalived/ scp /opt/k8s/conf/keepalived.service ${host}:/etc/systemd/system/ done for host in ${MASTER_IPS[@]} do printf "\e[1;34m${host}\e[0m\n" ssh root@${host} "systemctl daemon-reload && \ systemctl enable keepalived --now && \ systemctl status keepalived | grep Active" done
查看VIP所在的机器以及是否ping通
#!/usr/bin/env bash source /opt/k8s/bin/k8s-env.sh for host in ${MASTER_IPS[@]} do printf "\e[1;34m${host}\e[0m\n" ssh root@${host} "ip a | grep 192.168.72.100" done ping 192.168.72.100 -c 1
