34 篇文章5 订阅
Kubernetes高可用集群二进制部署（一）主机准备和负载均衡器安装
Kubernetes高可用集群二进制部署（二）ETCD集群部署
Kubernetes高可用集群二进制部署（三）部署api-server
Kubernetes高可用集群二进制部署（四）部署kubectl和kube-controller-manager、kube-scheduler
Kubernetes高可用集群二进制部署（五）kubelet、kube-proxy、Calico、CoreDNS
Kubernetes高可用集群二进制部署（六）Kubernetes集群节点添加
Kubernetes（简称为：k8s）是Google在2014年6月开源的一个容器集群管理系统，使用Go语言开发，用于管理云平台中多个主机上的容器化的应用，Kubernetes的目标是让部署容器化的应用简单并且高效,Kubernetes提供了资源调度、部署管理、服务发现、扩容缩容、监控，维护等一整套功能，努力成为跨主机集群的自动部署、扩展以及运行应用程序容器的平台。 它支持一系列容器工具, 包括Docker、Containerd等。
一、集群环境准备
1.1 主机规划
| 主机IP地址 | 主机名 | 主机配置 | 主机角色 | 软件列表 |
| 192.168.10.103 | k8s-master1 | 2C4G | master + worker | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce |
| 192.168.10.104 | k8s-master2 | 2C4G | master + worker | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce |
| 192.168.10.105 | k8s-master3 | 2C4G | master + worker | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce |
| 192.168.10.106 | k8s-worker1 | 2C4G | worker1 | kubelet、kube-proxy、docker-ce |
| 192.168.10.107 | k8s-worker2 | 2C4G | worker2(备用工作节点，集群搭建完成后作为新节点加入) | kubelet、kube-proxy、docker-ce |
| 192.168.10.101 | ha1 | 1C2G | LB | haproxy、keepalived |
| 192.168.10.102 | ha2 | 1C2G | LB | haproxy、keepalived |
| 192.168.10.100 | / | / | VIP(虚拟IP) |
1.2 软件版本
| 软件名称 | 版本 | 备注 |
| CentOS7 | kernel版本：5.16 | |
| kubernetes | v1.21.10 | |
| etcd | v3.5.2 | 最新版本 |
| calico | v3.19.4 | 网络插件 |
| coredns | v1.8.4 | |
| docker-ce | 20.10.13 | YUM源默认 |
| haproxy | 5.18 | YUM源默认 |
| keepalived | 3.5 | YUM源默认 |
1.3 网络分配
| 网络名称 | 网段 | 备注 |
| Node网络 | 192.168.10.101/107 | 集群节点网络 |
| Service网络 | 10.96.0.0/16 | 实现服务发现时所使用的网络 |
| Pod网络 | 10.244.0.0/16 |
二、集群部署
2.1主机准备
2.1.1 主机名设置
hostnamectl set-hostname xxx
关于主机名参见1.1小节主机规划表
2.1.2 主机与IP地址解析
cat >> /etc/hosts << EOF 192.168.10.101 ha1 192.168.10.102 ha2 192.168.10.103 k8s-master1 192.168.10.104 k8s-master2 192.168.10.105 k8s-master3 192.168.10.106 k8s-worker1 EOF
2.1.3 主机安全设置
2.1.3.1 关闭防火墙
systemctl stop firewalld systemctl disable firewalld firewall-cmd --state
2.1.3.2 关闭selinux
setenforce 0 sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config sestatus
2.1.4 交换分区设置
swapoff -a sed -ri 's/.*swap.*/#&/' /etc/fstab echo "vm.swappiness=0" >> /etc/sysctl.conf sysctl -p
2.1.5 主机系统时间同步
安装软件 yum -y install ntpdate 制定时间同步计划任务 crontab -e 0 */1 * * * ntpdate time1.aliyun.com
2.1.6 主机系统优化
limit优化
ulimit -SHn 65535
cat <<EOF >> /etc/security/limits.conf * soft nofile 655360 * hard nofile 131072 * soft nproc 655350 * hard nproc 655350 * soft memlock unlimited * hard memlock unlimited EOF
2.1.7 ipvs管理工具安装及模块加载
为集群节点安装，负载均衡节点不用安装
yum -y install ipvsadm ipset sysstat conntrack libseccom
所有节点配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可： modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack
创建 /etc/modules-load.d/ipvs.conf 并加入以下内容： cat >/etc/modules-load.d/ipvs.conf <<EOF ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp ip_vs_sh nf_conntrack ip_tables ip_set xt_set ipt_set ipt_rpfilter ipt_REJECT ipip EOF
设置为开机启动 systemctl enable --now systemd-modules-load.service
如果执行开机启动失败了，提示如下信息：
Job for systemd-modules-load.service failed because the control process exited with error code. See "systemctl status systemd-modules-load.service" and "journalctl -xe" for details. Failed to find module 'ip_vs_fo'
具体原因是内核版本问题，不过也可以将文件中的ip_vs_fo 去掉，然后继续执行
2.1.8 Linux内核升级
在所有节点中安装,需要重新操作系统更换内核。
[root@localhost ~]# yum -y install perl
[root@localhost ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
[root@localhost ~]# yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
[root@localhost ~]# yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64
[root@localhost ~]# grub2-set-default 0
[root@localhost ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
2.1.9 Linux内核优化
cat <<EOF > /etc/sysctl.d/k8s.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 fs.may_detach_mounts = 1 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 fs.file-max=52706963 fs.nr_open=52706963 net.netfilter.nf_conntrack_max=2310720 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_intvl =15 net.ipv4.tcp_max_tw_buckets = 36000 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_orphans = 327680 net.ipv4.tcp_orphan_retries = 3 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.ip_conntrack_max = 131072 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_timestamps = 0 net.core.somaxconn = 16384 EOF
sysctl --system
所有节点配置完内核后，重启服务器，保证重启后内核依旧加载 reboot -h now
重启后查看结果： lsmod | grep --color=auto -e ip_vs -e nf_conntrack
2.1.10 其它工具安装(选装)
yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz -y
2.2 负载均衡器准备
2.2.1 安装haproxy与keepalived
在负载均衡两台服务器上安装
yum -y install haproxy keepalived
2.2.2 HAProxy配置
cat >/etc/haproxy/haproxy.cfg<<"EOF" global maxconn 2000 ulimit-n 16384 log 127.0.0.1 local0 err stats timeout 30s defaults log global mode http option httplog timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s timeout http-keep-alive 15s frontend monitor-in bind *:33305 mode http option httplog monitor-uri /monitor frontend k8s-master bind 0.0.0.0:6443 bind 127.0.0.1:6443 mode tcp option tcplog tcp-request inspect-delay 5s default_backend k8s-master backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin #负载均衡策略 default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server k8s-master1 192.168.10.103:6443 check server k8s-master2 192.168.10.104:6443 check server k8s-master3 192.168.10.105:6443 check EOF
2.2.3 KeepAlived
主从配置不一致，需要注意。
KeepAlived主要是对haproxy进行监控
ha1: cat >/etc/keepalived/keepalived.conf<<"EOF" ! Configuration File for keepalived global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state MASTER #指定主节点 interface eth0 #网卡 mcast_src_ip 192.168.10.101 #本机ip virtual_router_id 51 priority 100 #优先级 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.10.100 #虚拟ip 主节点和备份节点一样的 } track_script { chk_apiserver #通过脚本实现监控 } } EOF
ha2: cat >/etc/keepalived/keepalived.conf<<"EOF" ! Configuration File for keepalived global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state BACKUP interface eth0 mcast_src_ip 192.168.10.102 virtual_router_id 51 priority 99 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.10.100 } track_script { chk_apiserver } } EOF
2.2.4 健康检查脚本
ha1及ha2均要配置（相同）
cat > /etc/keepalived/check_apiserver.sh <<"EOF" #!/bin/bash err=0 #定义变量 for k in $(seq 1 3) do check_code=$(pgrep haproxy) #检查haproxy进程 if [[ $check_code == "" ]]; then err=$(expr $err + 1) sleep 1 continue else err=0 break fi done if [[ $err != "0" ]]; then echo "systemctl stop keepalived" /usr/bin/systemctl stop keepalived exit 1 else exit 0 fi EOF
chmod +x /etc/keepalived/check_apiserver.sh
2.2.5 启动服务并验证
systemctl daemon-reload systemctl enable --now haproxy systemctl enable --now keepalived
ip address show
ss -anput | grep ":6443"
