Kubernetes高可用集群二进制部署（一）主机准备和负载均衡器安装
Kubernetes高可用集群二进制部署（二）ETCD集群部署
Kubernetes高可用集群二进制部署（三）部署api-server
Kubernetes高可用集群二进制部署（四）部署kubectl和kube-controller-manager、kube-scheduler
Kubernetes高可用集群二进制部署（五）kubelet、kube-proxy、Calico、CoreDNS
Kubernetes高可用集群二进制部署（六）Kubernetes集群节点添加
1. 配置免密登录
在k8s-master1上操作
生成密钥对，把公钥复制到远程主机
ssh-keygen #直接回车，不设置密码
ssh-copy-id root@k8s-master1 ssh-copy-id root@k8s-master2 ssh-copy-id root@k8s-master3 ssh-copy-id root@k8s-worker1
ssh root@k8s-master1 #依次此时是否能否正常登录
2 部署ETCD集群
在k8s-master1上操作。
2.1 创建工作目录
mkdir -p /data/k8s-work
2.2 获取cfssl工具
cd /data/k8s-work wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
说明： cfssl是使用go编写，由CloudFlare开源的一款PKI/TLS工具。主要程序有： - cfssl，是CFSSL的命令行工具 - cfssljson用来从cfssl程序获取JSON输出，并将证书，密钥，CSR和bundle写入文件中。
chmod +x cfssl*
mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
# cfssl version Version: 1.2.0 Revision: dev Runtime: go1.6
2.3 创建CA证书
在master1上创建证书
2.3.1 配置ca证书请求文件
在当前目录/data/k8s-work执行
cat > ca-csr.json <<"EOF" { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubemsb", "OU": "CN" } ], "ca": { "expiry": "87600h" } } EOF
2.3.2 创建ca证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2.3.3 配置ca证书策略
cat > ca-config.json <<"EOF" { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } EOF
server auth 表示client可以对使用该ca对server提供的证书进行验证 client auth 表示server可以使用该ca对client提供的证书进行验证
2.4 创建etcd证书
2.4.1 配置etcd请求文件
一般可以部署三台集群，可以是3、5、7、9
cat > etcd-csr.json <<"EOF" { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.10.103", "192.168.10.104", "192.168.10.105" ], "key": { "algo": "rsa", "size": 2048 }, "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubemsb", "OU": "CN" }] } EOF
2.4.2 生成etcd证书
确保在当前目录有ca-key.pem、ca-config.json、etcd-csr.json文件
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
# ls 输出 ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem etcd.csr etcd-csr.json etcd-key.pem etcd.pem
etcd.csr：etcd证书请求文件
etcd-key.pem：私钥
etcd.pem：基于私钥签发的证书
2.5 部署etcd集群
2.5.1 下载etcd软件包
wget https://github.com/etcd-io/etcd/releases/download/v3.5.2/etcd-v3.5.2-linux-amd64.tar.gz
2.5.2 安装etcd软件
tar -xvf etcd-v3.5.2-linux-amd64.tar.gz cp -p etcd-v3.5.2-linux-amd64/etcd* /usr/local/bin/ #-p 保留源文件的权限
2.5.3 分发etcd软件
scp etcd-v3.5.2-linux-amd64/etcd* k8s-master2:/usr/local/bin/ scp etcd-v3.5.2-linux-amd64/etcd* k8s-master3:/usr/local/bin/
2.5.4 创建配置文件
在三个etcd节点操作
mkdir /etc/etcd
其中地址都是当前主机的ip本身
cat > /etc/etcd/etcd.conf <<"EOF" #[Member] ETCD_NAME="etcd1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.10.103:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.10.103:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.103:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.103:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.103:2380,etcd2=https://192.168.10.104:2380,etcd3=https://192.168.10.105:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" EOF
说明： ETCD_NAME：节点名称，集群中唯一 ETCD_DATA_DIR：数据目录 ETCD_LISTEN_PEER_URLS：集群通信监听地址 ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址 ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址 ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址 ETCD_INITIAL_CLUSTER：集群节点地址 ETCD_INITIAL_CLUSTER_TOKEN：集群Token ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群
2.5.5 创建服务配置文件
mkdir -p /etc/etcd/ssl mkdir -p /var/lib/etcd/default.etcd
cd /data/k8s-work cp ca*.pem /etc/etcd/ssl cp etcd*.pem /etc/etcd/ssl
cat > /etc/systemd/system/etcd.service <<"EOF" [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=-/etc/etcd/etcd.conf WorkingDirectory=/var/lib/etcd/ ExecStart=/usr/local/bin/etcd \ --cert-file=/etc/etcd/ssl/etcd.pem \ --key-file=/etc/etcd/ssl/etcd-key.pem \ --trusted-ca-file=/etc/etcd/ssl/ca.pem \ --peer-cert-file=/etc/etcd/ssl/etcd.pem \ --peer-key-file=/etc/etcd/ssl/etcd-key.pem \ --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \ --peer-client-cert-auth \ --client-cert-auth Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
2.5.6 同步etcd配置到集群其它master节点
创建目录 mkdir -p /etc/etcd mkdir -p /etc/etcd/ssl mkdir -p /var/lib/etcd/default.etcd
服务配置文件,需要修改etcd节点名称及IP地址 for i in k8s-master2 k8s-master3 do scp /etc/etcd/etcd.conf $i:/etc/etcd/ done
k8s-master2: cat /etc/etcd/etcd.conf #[Member] ETCD_NAME="etcd2" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.10.104:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.10.104:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.104:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.104:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.103:2380,etcd2=https://192.168.10.104:2380,etcd3=https://192.168.10.105:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"
k8s-master3: cat /etc/etcd/etcd.conf #[Member] ETCD_NAME="etcd3" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.10.105:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.10.105:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.105:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.105:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.103:2380,etcd2=https://192.168.10.104:2380,etcd3=https://192.168.10.105:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"
证书文件 for i in k8s-master2 k8s-master3 do scp /etc/etcd/ssl/* $i:/etc/etcd/ssl done
服务启动配置文件 for i in k8s-master2 k8s-master3 do scp /etc/systemd/system/etcd.service $i:/etc/systemd/system/ done
2.5.7 启动etcd集群
三台主机全都执行
systemctl daemon-reload systemctl enable --now etcd.service #现在开启并设置开机自启 systemctl status etcd
2.5.8 验证集群状态
ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.10.103:2379,https://192.168.10.104:2379,https://192.168.10.105:2379 endpoint health
+-----------------------------+--------+------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +-----------------------------+--------+------------+-------+ | https://192.168.10.103:2379 | true | 6.961777ms | | | https://192.168.10.105:2379 | true | 6.594067ms | | | https://192.168.10.104:2379 | true | 6.835899ms | | +-----------------------------+--------+------------+-------+
耗时较长的话可以做一些优化工作
