部署 scheduler 组件
配置 scheduler 证书
下面的 ip 记得替换成自己的节点 ip
- 和 etcd 组件一样，把所有的 scheduler 节点 ip 都要写进去
- 同样的，在配置 ip 的时候，注意 json 格式的
,逗号位置cat << EOF > ${work_dir}/tmp/ssl/kube-scheduler-csr.json { "CN": "system:kube-scheduler", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.91.19" ], "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "system:kube-scheduler", "OU": "System" } ] } EOF
创建 scheduler 证书
设置集群参数
--server为 apiserver 的访问地址（如果有高可用地址，一定要写高可用的地址）- 修改成自己的 ip 地址和 service 文件里面指定的
--secure-port参数的端口（如果是高可用地址，要写高可用的端口）- 切记，一定要带上
https://协议，否则生成的证书，kubectl命令访问不到 apiserver
${work_dir}/bin/cfssl gencert \ -ca=${work_dir}/tmp/ssl/ca.pem \ -ca-key=${work_dir}/tmp/ssl/ca-key.pem \ -config=${work_dir}/tmp/ssl/ca-config.json \ -profile=kubernetes ${work_dir}/tmp/ssl/kube-scheduler-csr.json | \ ${work_dir}/bin/cfssljson -bare ${work_dir}/tmp/ssl/kube-scheduler
创建 kubeconfig 证书
设置集群参数
--server为 apiserver 的访问地址（如果有高可用地址，一定要写高可用的地址）
- 修改成自己的 ip 地址和 service 文件里面指定的
--secure-port参数的端口（如果是高可用地址，要写高可用的端口） - 切记，一定要带上
https://协议，否则生成的证书，kubectl命令访问不到 apiserver
${work_dir}/bin/kubernetes/server/bin/kubectl config \ set-cluster kubernetes \ --certificate-authority=${work_dir}/tmp/ssl/ca.pem \ --embed-certs=true \ --server=https://192.168.11.147:6443 \ --kubeconfig=${work_dir}/tmp/ssl/kube-scheduler.kubeconfig
设置客户端认证参数
${work_dir}/bin/kubernetes/server/bin/kubectl config \ set-credentials system:kube-scheduler \ --client-certificate=${work_dir}/tmp/ssl/kube-scheduler.pem \ --client-key=${work_dir}/tmp/ssl/kube-scheduler-key.pem \ --embed-certs=true \ --kubeconfig=${work_dir}/tmp/ssl/kube-scheduler.kubeconfig
设置上下文参数
${work_dir}/bin/kubernetes/server/bin/kubectl config \ set-context system:kube-scheduler \ --cluster=kubernetes \ --user=system:kube-scheduler \ --kubeconfig=${work_dir}/tmp/ssl/kube-scheduler.kubeconfig
设置默认上下文
${work_dir}/bin/kubernetes/server/bin/kubectl config \ use-context system:kube-scheduler \ --kubeconfig=${work_dir}/tmp/ssl/kube-scheduler.kubeconfig
配置 scheduler 为 systemctl 管理
创建 systemctl 启动模板
cat << EOF > ${work_dir}/tmp/service/kube-scheduler.service.template [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] ExecStart=##k8sBin##/kube-scheduler \\ --authentication-kubeconfig=##configPath##/kube-scheduler.kubeconfig \\ --authorization-kubeconfig=##configPath##/kube-scheduler.kubeconfig \\ --bind-address=0.0.0.0 \\ --kubeconfig=##configPath##/kube-scheduler.kubeconfig \\ --leader-elect=true \\ --v=2 Restart=always RestartSec=5 [Install] WantedBy=multi-user.target EOF
生成各个 scheduler 节点的 systemctl 启动文件
k8sBin='/data/kubernetes/bin'; \ sslPath='/etc/kubernetest/ssl'; \ configPath='/etc/kubernetest'; \ sed -e "s|##k8sBin##|${k8sBin}|g" \ -e "s|##sslPath##|${sslPath}|g" \ -e "s|##configPath##|${configPath}|g" ${work_dir}/tmp/service/kube-scheduler.service.template \ > ${work_dir}/tmp/service/kube-scheduler.service
分发文件并启动 scheduler 集群
下面的 ip 记得替换成自己的节点 ip
ip_head='192.168.11';for i in 147 148 149;do \ k8sHost="${ip_head}.${i}"; \ ssh ${k8sHost} "mkdir -p ${k8sBin} ${sslPath}"; \ scp ${work_dir}/tmp/ssl/kube-scheduler.kubeconfig ${k8sHost}:${configPath}/; \ scp ${work_dir}/tmp/service/kube-scheduler.service ${k8sHost}:/usr/lib/systemd/system/; \ scp ${work_dir}/bin/kubernetes/server/bin/kube-scheduler ${k8sHost}:${k8sBin}/; \ ssh ${k8sHost} "systemctl enable kube-scheduler && systemctl start kube-scheduler --no-block"; \ done
到这里，master 节点的部署就结束了，可以使用下面的命令查看 master 组件的状态
${work_dir}/bin/kubernetes/server/bin/kubectl get componentstatus
可以看到状态都是 Healthy，下面的 Warning 只是说 ComponentStatus 在 v1 版本的 api 里面，从 1.19 以后的版本取消了
Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true","reason":""} etcd-2 Healthy {"health":"true","reason":""} etcd-1 Healthy {"health":"true","reason":""}
部署 worker 组件
部署 docker 组件
准备 docker 二进制文件
我提前下载好了 docker 二进制文件，放在了
${work_dir}/bin目录下了，大家要注意自己放的路径
cd ${work_dir}/bin && \ tar xf docker-23.0.1.tgz
配置 docker 为 systemctl 管理
创建 systemctl 启动模板
cat <<EOF > ${work_dir}/tmp/service/docker.service.template [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target [Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker ExecStart=/usr/bin/dockerd \\ -H unix://var/run/docker.sock \\ --data-root=##dataRoot## ExecReload=/bin/kill -s HUP $MAINPID # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity # Uncomment TasksMax if your systemd version supports it. # Only systemd 226 and above support this version. #TasksMax=infinity TimeoutStartSec=0 # set delegate yes so that systemd does not reset the cgroups of docker containers Delegate=yes # kill only the docker process, not all processes in the cgroup KillMode=process # restart the docker process if it exits prematurely Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target EOF
准备 daemon.json 文件
cat <<EOF > ${work_dir}/tmp/service/daemon.json.template { "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"], "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ] } EOF
分发文件并启动 docker 组件
下面的 ip 记得替换成自己的节点 ip
ip_head='192.168.11';for i in 147 148 149;do \ dataRoot='/data/docker'; \ sed "s|##dataRoot##|${dataRoot}|g" ${work_dir}/tmp/service/docker.service.template \ > ${work_dir}/tmp/service/docker.service; \ ssh ${ip_head}.${i} "mkdir -p /etc/docker"; \ scp ${work_dir}/tmp/service/docker.service ${ip_head}.${i}:/usr/lib/systemd/system/; \ scp ${work_dir}/tmp/service/daemon.json.template ${ip_head}.${i}:/etc/docker/daemon.json; \ scp ${work_dir}/bin/docker/* ${ip_head}.${i}:/usr/bin/; \ ssh ${ip_head}.${i} "systemctl enable docker && systemctl start docker --no-block"; \ done
部署 cri-dockerd 组件
准备 cri-dockerd 二进制文件
我提前下载好了 cri-dockerd 二进制文件，放在了
${work_dir}/bin目录下了，大家要注意自己放的路径
cd ${work_dir}/bin && \ tar xf cri-dockerd-0.3.1.amd64.tgz
配置 cri-dockerd 为 systemctl 管理
- 官方给出的模板，一些参数是需要做修改的
--cni-bin-dir默认路径是/opt/cni/bin，有需要的话，可以做修改，为了避免大家后面忘记了，这里我就不做修改了，使用默认的路径--container-runtime-endpoint默认的路径是unix:///var/run/cri-dockerd.sock，可以把官方的这个参数去掉就好--cri-dockerd-root-directory默认路径是/var/lib/cri-dockerd，可以和 docker 一样修改路径，避免默认的路径空间不足
--pod-infra-container-image默认镜像是registry.k8s.io/pause:3.6，要改成阿里的镜像，如果没有国外服务器，拉取 k8s 的镜像会失败
- 其他参数可以通过
cri-dockerd --help命令来获取
创建 systemctl 启动模板
cat <<EOF > ${work_dir}/tmp/service/cri-dockerd.service.template [Unit] Description=CRI Interface for Docker Application Container Engine Documentation=https://docs.mirantis.com After=network-online.target firewalld.service docker.service Wants=network-online.target # 懒得配置 cri-docker.socket 了，我这边就直接注释掉了，不然启动会报错：Unit not found # Requires=cri-docker.socket [Service] Type=notify ExecStart=/usr/bin/cri-dockerd \\ --cri-dockerd-root-directory=##criData## \\ --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9 ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always # Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229. # Both the old, and new location are accepted by systemd 229 and up, so using the old location # to make them work for either version of systemd. StartLimitBurst=3 # Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230. # Both the old, and new name are accepted by systemd 230 and up, so using the old name to make # this option work for either version of systemd. StartLimitInterval=60s # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity # Comment TasksMax if your systemd version does not support it. # Only systemd 226 and above support this option. TasksMax=infinity Delegate=yes KillMode=process [Install] WantedBy=multi-user.target EOF
分发文件并启动 cri-dockerd 组件
下面的 ip 记得替换成自己的节点 ip
- 下面的操作，需要在有网络的情况下执行
- 会通过 yum 安装 conntrack-tools
subnet参数要和controller-manager的--cluster-cidr参数一致，注意修改subNet变量的值- 我的 cri-dockerd 没有改 cni 插件地址，默认是
/opt/cni/bin，如果大家有修改的话，下面的cni_dir变量值记得替换- 我的 cri-dockerd 没有改 cni 网络配置文件的地址，默认是
/etc/cni/net.d，如果大家有修改的话，下面的cni_conf变量值记得替换ip_head='192.168.11';for i in 147 148 149;do \ criData='/data/cri-dockerd'; \ sed "s|##criData##|${criData}|g" ${work_dir}/tmp/service/cri-dockerd.service.template \ > ${work_dir}/tmp/service/cri-dockerd.service; \ scp ${work_dir}/tmp/service/cri-dockerd.service ${ip_head}.${i}:/usr/lib/systemd/system/; \ scp ${work_dir}/bin/cri-dockerd/* ${ip_head}.${i}:/usr/bin/; \ ssh ${ip_head}.${i} "yum install -y conntrack-tools"; \ ssh ${ip_head}.${i} "systemctl enable cri-dockerd && systemctl start cri-dockerd --no-block"; \ done
部署 kubelet 组件
配置 kubelet 证书
cat << EOF > ${work_dir}/tmp/ssl/kubelet-csr.json.template { "CN": "system:node:##nodeHost##", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "##nodeHost##" ], "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "system:nodes", "OU": "System" } ] } EOF
创建 kubelet 证书
下面的 ip 记得替换成自己的节点 ip
ip_head='192.168.11';for i in 147 148 149;do \ nodeHost="${ip_head}.${i}"; \ sed "s|##nodeHost##|${nodeHost}|g" ${work_dir}/tmp/ssl/kubelet-csr.json.template \ > ${work_dir}/tmp/ssl/kubelet-csr.${nodeHost}.json ${work_dir}/bin/cfssl gencert \ -ca=${work_dir}/tmp/ssl/ca.pem \ -ca-key=${work_dir}/tmp/ssl/ca-key.pem \ -config=${work_dir}/tmp/ssl/ca-config.json \ -profile=kubernetes ${work_dir}/tmp/ssl/kubelet-csr.${nodeHost}.json | \ ${work_dir}/bin/cfssljson -bare ${work_dir}/tmp/ssl/kubelet.${nodeHost}; done
创建 kubeconfig 证书
设置集群参数
--server为 apiserver 的访问地址（如果有高可用地址，一定要写高可用的地址）
- 修改成自己的 ip 地址和 service 文件里面指定的
--secure-port参数的端口（如果是高可用地址，要写高可用的端口） - 切记，一定要带上
https://协议，否则生成的证书，kubectl命令访问不到 apiserver
ip_head='192.168.11';for i in 147 148 149;do \ nodeHost="${ip_head}.${i}"; \ ${work_dir}/bin/kubernetes/server/bin/kubectl config \ set-cluster kubernetes \ --certificate-authority=${work_dir}/tmp/ssl/ca.pem \ --embed-certs=true \ --server=https://192.168.11.147:6443 \ --kubeconfig=${work_dir}/tmp/ssl/kubelet.kubeconfig.${nodeHost}; done
设置客户端认证参数
ip_head='192.168.11';for i in 147 148 149;do \ nodeHost="${ip_head}.${i}"; \ ${work_dir}/bin/kubernetes/server/bin/kubectl config \ set-credentials system:node:${nodeHost} \ --client-certificate=${work_dir}/tmp/ssl/kubelet.${nodeHost}.pem \ --client-key=${work_dir}/tmp/ssl/kubelet.${nodeHost}-key.pem \ --embed-certs=true \ --kubeconfig=${work_dir}/tmp/ssl/kubelet.kubeconfig.${nodeHost}; done
设置上下文参数
ip_head='192.168.11';for i in 147 148 149;do \ nodeHost="${ip_head}.${i}"; \ ${work_dir}/bin/kubernetes/server/bin/kubectl config \ set-context system:node:${nodeHost} \ --cluster=kubernetes \ --user=system:node:${nodeHost} \ --kubeconfig=${work_dir}/tmp/ssl/kubelet.kubeconfig.${nodeHost}; done
设置默认上下文
ip_head='192.168.11';for i in 147 148 149;do \ nodeHost="${ip_head}.${i}"; \ ${work_dir}/bin/kubernetes/server/bin/kubectl config \ use-context system:node:${nodeHost} \ --kubeconfig=${work_dir}/tmp/ssl/kubelet.kubeconfig.${nodeHost}; done
配置 kubelet 配置文件
clusterDNS参数的 ip 注意修改，和apiserver的--service-cluster-ip-range参数一个网
段，和 k8s 服务 ip 要不一样，一般 k8s 服务的 ip 取网段第一个ip， clusterdns 选网段的第二个ip
cat << EOF > ${work_dir}/tmp/service/kubelet-config.yaml.template kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 0.0.0.0 authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: ##sslPath##/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s cgroupDriver: systemd cgroupsPerQOS: true clusterDNS: - 10.88.0.2 clusterDomain: cluster.local configMapAndSecretChangeDetectionStrategy: Watch containerLogMaxFiles: 3 containerLogMaxSize: 10Mi enforceNodeAllocatable: - pods eventBurst: 10 eventRecordQPS: 5 evictionHard: imagefs.available: 15% memory.available: 300Mi nodefs.available: 10% nodefs.inodesFree: 5% evictionPressureTransitionPeriod: 5m0s failSwapOn: true fileCheckFrequency: 40s hairpinMode: hairpin-veth healthzBindAddress: 0.0.0.0 healthzPort: 10248 httpCheckFrequency: 40s imageGCHighThresholdPercent: 85 imageGCLowThresholdPercent: 80 imageMinimumGCAge: 2m0s kubeAPIBurst: 100 kubeAPIQPS: 50 makeIPTablesUtilChains: true maxOpenFiles: 1000000 maxPods: 110 nodeLeaseDurationSeconds: 40 nodeStatusReportFrequency: 1m0s nodeStatusUpdateFrequency: 10s oomScoreAdj: -999 podPidsLimit: -1 port: 10250 # disable readOnlyPort readOnlyPort: 0 resolvConf: /etc/resolv.conf runtimeRequestTimeout: 2m0s serializeImagePulls: true streamingConnectionIdleTimeout: 4h0m0s syncFrequency: 1m0s tlsCertFile: ##sslPath##/kubelet.pem tlsPrivateKeyFile: ##sslPath##/kubelet-key.pem EOF
配置 kubelet 为 systemctl 管理
--container-runtime参数默认是 docker ，如果使用 docker 以外的，需要配置为remote，并且要配置--container-runtime-endpoint参数来指定 sock 文件的路径- 我们这里需要改成
/var/run/cri-dockerd.sock来指定使用 cri-dockerd
--pod-infra-container-image参数和 cri-dockerd 的一样，改成阿里镜像的创建 systemctl 启动模板
cat << EOF > ${work_dir}/tmp/service/kubelet.service.template [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] WorkingDirectory=##dataRoot##/kubelet ExecStart=##k8sBin##/kubelet \\ --config=##dataRoot##/kubelet/kubelet-config.yaml \\ --container-runtime=remote \\ --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock \\ --hostname-override=##nodeHost## \\ --kubeconfig=##configPath##/kubelet.kubeconfig \\ --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9 \\ --root-dir=##dataRoot##/kubelet \\ --v=2 Restart=always RestartSec=5 [Install] WantedBy=multi-user.target EOF
分发文件并启动 kubelet 组件
下面的 ip 记得替换成自己的节点 ip
ip_head='192.168.11';for i in 147 148 149;do \ nodeHost="${ip_head}.${i}"; \ k8sBin='/data/kubernetes/bin'; \ sslPath='/etc/kubernetest/ssl'; \ configPath='/etc/kubernetest'; \ dataRoot='/data/kubernetes/data' sed "s|##sslPath##|${sslPath}|g" ${work_dir}/tmp/service/kubelet-config.yaml.template \ > ${work_dir}/tmp/service/kubelet-config.yaml; sed -e "s|##dataRoot##|${dataRoot}|g" \ -e "s|##k8sBin##|${k8sBin}|g" \ -e "s|##configPath##|${configPath}|g" \ -e "s|##nodeHost##|${nodeHost}|g" ${work_dir}/tmp/service/kubelet.service.template \ > ${work_dir}/tmp/service/kubelet.service.${nodeHost} ssh ${nodeHost} "mkdir -p ${k8sBin} ${sslPath} ${configPath} ${dataRoot}/kubelet"; \ scp ${work_dir}/tmp/ssl/ca*.pem ${nodeHost}:${sslPath}/; \ scp ${work_dir}/tmp/ssl/kubelet.${nodeHost}.pem ${nodeHost}:${sslPath}/kubelet.pem; \ scp ${work_dir}/tmp/ssl/kubelet.${nodeHost}-key.pem ${nodeHost}:${sslPath}/kubelet-key.pem; \ scp ${work_dir}/tmp/ssl/kubelet.kubeconfig.${nodeHost} ${nodeHost}:${configPath}/kubelet.kubeconfig; \ scp ${work_dir}/tmp/service/kubelet.service.${nodeHost} ${nodeHost}:/usr/lib/systemd/system/kubelet.service; \ scp ${work_dir}/tmp/service/kubelet-config.yaml ${nodeHost}:${dataRoot}/kubelet/kubelet-config.yaml; \ scp ${work_dir}/bin/kubernetes/server/bin/kubelet ${nodeHost}:${k8sBin}/; \ ssh ${nodeHost} "systemctl enable kubelet && systemctl start kubelet --no-block"; \ done
部署 proxy 组件
配置 proxy 证书
cat << EOF > ${work_dir}/tmp/ssl/kube-proxy-csr.json { "CN": "system:kube-proxy", "key": { "algo": "rsa", "size": 2048 }, "hosts": [], "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "system:kube-proxy", "OU": "System" } ] } EOF
创建 proxy 证书
创建 kubeconfig 证书
设置集群参数
--server为 apiserver 的访问地址（如果有高可用地址，一定要写高可用的地址）
- 修改成自己的 ip 地址和 service 文件里面指定的
--secure-port参数的端口（如果是高可用地址，要写高可用的端口） - 切记，一定要带上
https://协议，否则生成的证书，kubectl命令访问不到 apiserver
${work_dir}/bin/kubernetes/server/bin/kubectl config \ set-cluster kubernetes \ --certificate-authority=${work_dir}/tmp/ssl/ca.pem \ --embed-certs=true \ --server=https://192.168.11.147:6443 \ --kubeconfig=${work_dir}/tmp/ssl/kube-proxy.kubeconfig
设置客户端认证参数
${work_dir}/bin/kubernetes/server/bin/kubectl config \ set-credentials system:kube-proxy \ --client-certificate=${work_dir}/tmp/ssl/kube-proxy.pem \ --client-key=${work_dir}/tmp/ssl/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=${work_dir}/tmp/ssl/kube-proxy.kubeconfig
设置上下文参数
${work_dir}/bin/kubernetes/server/bin/kubectl config \ set-context system:kube-proxy \ --cluster=kubernetes \ --user=system:kube-proxy \ --kubeconfig=${work_dir}/tmp/ssl/kube-proxy.kubeconfig
设置默认上下文
${work_dir}/bin/kubernetes/server/bin/kubectl config \ use-context system:kube-proxy \ --kubeconfig=${work_dir}/tmp/ssl/kube-proxy.kubeconfig
配置 proxy 配置文件
clusterCIDR参数要和controller-manager的--cluster-cidr参数一致
hostnameOverride要和kubelet的--hostname-override参数一致，否则会出现node not found的报错
cat << EOF > ${work_dir}/tmp/service/kube-proxy-config.yaml.template kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 clientConnection: kubeconfig: "##configPath##/kube-proxy.kubeconfig" clusterCIDR: "172.20.0.0/16" conntrack: maxPerCore: 32768 min: 131072 tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s healthzBindAddress: 0.0.0.0:10256 hostnameOverride: "##nodeHost##" metricsBindAddress: 0.0.0.0:10249 mode: "ipvs" EOF
配置 proxy 为 systemctl 管理
cat << EOF > ${work_dir}/tmp/service/kube-proxy.service.template [Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] # kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量 ## 指定 --cluster-cidr 或 --masquerade-all 选项后 ## kube-proxy 会对访问 Service IP 的请求做 SNAT WorkingDirectory=##dataRoot##/kube-proxy ExecStart=##k8sBin##/kube-proxy \\ --config=##dataRoot##/kube-proxy/kube-proxy-config.yaml Restart=always RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
分发文件并启动 proxy 组件
下面的 ip 记得替换成自己的节点 ip
ip_head='192.168.11';for i in 147 148 149;do \ nodeHost="${ip_head}.${i}"; \ k8sBin='/data/kubernetes/bin'; \ configPath='/etc/kubernetest'; \ dataRoot='/data/kubernetes/data' sed -e "s|##configPath##|${configPath}|g" \ -e "s|##nodeHost##|${nodeHost}|g" ${work_dir}/tmp/service/kube-proxy-config.yaml.template \ > ${work_dir}/tmp/service/kube-proxy-config.yaml.${nodeHost}; sed -e "s|##dataRoot##|${dataRoot}|g" \ -e "s|##k8sBin##|${k8sBin}|g" ${work_dir}/tmp/service/kube-proxy.service.template \ > ${work_dir}/tmp/service/kube-proxy.service ssh ${nodeHost} "mkdir -p ${k8sBin} ${configPath} ${dataRoot}/kube-proxy"; \ scp ${work_dir}/tmp/ssl/kube-proxy.kubeconfig ${nodeHost}:${configPath}/; \ scp ${work_dir}/tmp/service/kube-proxy.service ${nodeHost}:/usr/lib/systemd/system/; \ scp ${work_dir}/tmp/service/kube-proxy-config.yaml.${nodeHost} ${nodeHost}:${dataRoot}/kube-proxy/kube-proxy-config.yaml; \ scp ${work_dir}/bin/kubernetes/server/bin/kube-proxy ${nodeHost}:${k8sBin}/; \ ssh ${nodeHost} "systemctl enable kube-proxy && systemctl start kube-proxy --no-block"; \ done
部署 calico 网络插件
下载 calico 的 yaml 文件
wget -O ${work_dir}/calico.yaml --no-check-certificate https://docs.tigera.io/archive/v3.25/manifests/calico.yaml
配置 cidr 网段，找到 CALICO_IPV4POOL_CIDR 字段，关闭前面的注释，把 ip 网段修改成和 controller-manager 的 --cluster-cidr 参数一致
- name: CALICO_IPV4POOL_CIDR value: "172.20.0.0/16"
创建 calico 组件
${work_dir}/bin/kubernetes/server/bin/kubectl apply -f ${work_dir}/calico.yaml
部署 coredns 组件
clusterIP参数要和kubelet配置文件的clusterDNS参数一致
cat << EOF > ${work_dir}/coredns.yaml apiVersion: v1 kind: ServiceAccount metadata: name: coredns namespace: kube-system labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: kubernetes.io/bootstrapping: rbac-defaults addonmanager.kubernetes.io/mode: Reconcile name: system:coredns rules: - apiGroups: - "" resources: - endpoints - services - pods - namespaces verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults addonmanager.kubernetes.io/mode: EnsureExists name: system:coredns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:coredns subjects: - kind: ServiceAccount name: coredns namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system labels: addonmanager.kubernetes.io/mode: EnsureExists data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 reload loadbalance } --- apiVersion: apps/v1 kind: Deployment metadata: name: coredns namespace: kube-system labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS" spec: replicas: 1 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 selector: matchLabels: k8s-app: kube-dns template: metadata: labels: k8s-app: kube-dns spec: securityContext: seccompProfile: type: RuntimeDefault priorityClassName: system-cluster-critical serviceAccountName: coredns affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: k8s-app operator: In values: ["kube-dns"] topologyKey: kubernetes.io/hostname tolerations: - key: "CriticalAddonsOnly" operator: "Exists" nodeSelector: kubernetes.io/os: linux containers: - name: coredns image: coredns/coredns:1.9.3 imagePullPolicy: IfNotPresent resources: limits: memory: 300Mi requests: cpu: 100m memory: 70Mi args: [ "-conf", "/etc/coredns/Corefile" ] volumeMounts: - name: config-volume mountPath: /etc/coredns readOnly: true ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP - containerPort: 9153 name: metrics protocol: TCP livenessProbe: httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: httpGet: path: /ready port: 8181 scheme: HTTP securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - all readOnlyRootFilesystem: true dnsPolicy: Default volumes: - name: config-volume configMap: name: coredns items: - key: Corefile path: Corefile --- apiVersion: v1 kind: Service metadata: name: kube-dns namespace: kube-system annotations: prometheus.io/port: "9153" prometheus.io/scrape: "true" labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS" spec: selector: k8s-app: kube-dns clusterIP: 10.88.0.2 ports: - name: dns port: 53 protocol: UDP - name: dns-tcp port: 53 protocol: TCP - name: metrics port: 9153 protocol: TCP EOF
部署 coredns 组件
${work_dir}/bin/kubernetes/server/bin/kubectl apply -f ${work_dir}/coredns.yaml
测试 coredns
cat<<EOF | ${work_dir}/bin/kubernetes/server/bin/kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: containers: - name: busybox image: busybox:1.28.3 command: - sleep - "3600" imagePullPolicy: IfNotPresent restartPolicy: Always EOF
使用 busybox 容器内的 nslookup 命令来解析 kubernetes 域名
${work_dir}/bin/kubernetes/server/bin/kubectl exec busybox -- nslookup kubernetes
返回结果类似如下，表示解析没问题
Server: 10.88.0.2 Address 1: 10.88.0.2 kube-dns.kube-system.svc.cluster.local Name: kubernetes Address 1: 10.88.0.1 kubernetes.default.svc.cluster.local
