生成证书：
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
总共会生成4个证书文件，这四个文件是以pem为后缀的，将这四个文件拷贝到 /opt/kubernetes/ssl 目录下：
cp server*.pem ca*.pem /opt/kubernetes/ssl/
证书生成的工作就到这告一段落了。下面是启用 TLS Bootstrapping 自签机制。
cat > /opt/kubernetes/cfg/token.csv << EOF c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:nodebootstrapper" EOF
这里的token可以使用下面的命令生成然后替换：
head -c 16 /dev/urandom | od -An -t x | tr -d ' '
kube-apiserver的启动脚本：
vim /usr/lib/systemd/system/kube-apiserver.service
[Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target
服务启动和加入自启：
1. systemctl daemon-reload 2. systemctl start kube-apiserver 3. systemctl enable kube-apiserver
该服务状态为绿色表示正常：
[root@master ssl]# systemctl status kube-apiserver ● kube-apiserver.service - Kubernetes API Server Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2022-08-26 15:33:19 CST; 6h ago Docs: https://github.com/kubernetes/kubernetes Main PID: 3009 (kube-apiserver) Memory: 365.3M CGroup: /system.slice/kube-apiserver.service └─3009 /opt/kubernetes/bin/kube-apiserver --v=2 --logtostderr=false --log-dir=/opt/kubernetes/logs --etcd-servers=https://192.168.217.16:2379,https://1... Aug 26 15:33:19 master systemd[1]: Started Kubernetes API Server. Aug 26 15:33:19 master systemd[1]: Starting Kubernetes API Server... Aug 26 15:33:28 master kube-apiserver[3009]: E0826 15:33:28.034854 3009 controller.go:152] Unable to remove old endpoints from kubernetes service: Sto...ErrorMsg: Hint: Some lines were ellipsized, use -l to show in full.
如果有错误导致服务未能正常启动，可查看系统日志 /var/log/messages
通过对/var/log/messages日志的观察，可以发现，在第一次启动apiserver的时候，生成了非常多的角色，这些角色对应了k8s内的各种资源。例如：
Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.321342 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/cluster-admin Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.335178 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:discovery Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.346905 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:basic-user Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.359675 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:public-info-viewer Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.370449 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/admin Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.381805 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/edit Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.395624 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/view Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.406568 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-admin Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.415029 6822 healthz.go:200] [+]ping ok
Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.516294 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:kube-aggregator Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.525808 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:kube-controller-manager Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.535778 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:kube-dns Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.545944 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:persistent-volume-provisioner Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.558356 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:nodeclient Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.567806 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.577033 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:volume-scheduler Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.585929 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:legacy-unknown-approver Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.596499 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:kubelet-serving-approver Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.605861 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:ku
g 30 10:01:23 master kube-apiserver: I0830 10:01:23.614996 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:kube-apiserver-client-kubelet-approver Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.624625 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:node-proxier Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.635380 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:kube-scheduler Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.644132 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:attachdetach-controller Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.653821 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:clusterrole-aggregation-controller Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.663108 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:cronjob-controller Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.672682 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:daemon-set-controller Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.685326 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:deployment-controller Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.694401 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:disruption-controller Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.703354 6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:endpoint-controller Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.713226 6822 healthz.go:200] [+]ping ok Aug 30 10:01:23 master kube-apiserver: [+]log ok Aug 30 10:01:23 master kube-apiserver: [+]etcd ok
Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.123145 6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:endpointslice-controller Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.132424 6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:expand-controller Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.149014 6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:generic-garbage-collector Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.160210 6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:horizontal-pod-autoscaler Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.169018 6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:job-controller Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.178514 6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:namespace-controller Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.187484 6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:node-controller Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.201137 6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:persistent-volume-binder Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.213896 6822 healthz.go:200] [+]ping ok
（9）部署kube-controller-manager
该服务的配置文件：
vim /opt/kubernetes/cfg/kube-controller-manager.conf
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --leader-elect=true \ --master=127.0.0.1:8080 \ --bind-address=127.0.0.1 \ --allocate-node-cidrs=true \ --cluster-cidr=10.244.0.0/16 \ --service-cluster-ip-range=10.0.0.0/16 \ --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \ --root-ca-file=/opt/kubernetes/ssl/ca.pem \ --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \ --experimental-cluster-signing-duration=87600h0m0s"
配置文件说明：
--master：通过本地非安全本地端口8080连接apiserver。 --leader-elect：当该组件启动多个时，自动选举（HA） --cluster-signing-cert-file/--cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致，也就是两个服务共用ca证书。
该服务的启动脚本：
vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target
启动并设置开机启动：
1. systemctl daemon-reload 2. systemctl start kube-controller-manager 3. systemctl enable kube-controller-manager
（10）部署kube-scheduler
这个服务是调度服务，主要调度各类资源的，通过和controller-manage服务通信，以及etcd通知进行各类资源调度。
配置文件：
vim /opt/kubernetes/cfg/kube-scheduler.conf
KUBE_SCHEDULER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --leader-elect \ --master=127.0.0.1:8080 \ --bind-address=127.0.0.1"
配置文件说明：
1. --master：通过本地非安全本地端口8080连接apiserver。 2. --leader-elect：当该组件启动多个时，自动选举（HA）
启动脚本：
vim /usr/lib/systemd/system/kube-scheduler.service
[Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target
启动并设置开机启动：
1. systemctl daemon-reload 2. systemctl start kube-scheduler 3. systemctl enable kube-scheduler
（11）
此时，这三个服务搭建完毕后，就可以集群的健康检查了：
[root@master cfg]# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health":"true"} etcd-1 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"}
如果哪个服务没有启动或者异常，此命令都会显示出来。例如，停止一个etcd，上面的命令将会报告错误：
[root@master cfg]# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok etcd-1 Unhealthy Get https://192.168.217.17:2379/health: dial tcp 192.168.217.17:2379: connect: connection refused controller-manager Healthy ok etcd-0 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"}
（12）node节点安装kubelet
kubelet服务是node工作节点比较重要的一个服务，这个服务也不太好配置：
kubelet服务的配置文件：
vim /opt/kubernetes/cfg/kubelet.conf
KUBELET_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --hostname-override=k8s-master \ --network-plugin=cni \ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \ --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \ --config=/opt/kubernetes/cfg/kubelet-config.yml \ --cert-dir=/opt/kubernetes/ssl \ --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2"
导入相关镜像包，包名是registry.cn-hangzhou.aliyuncs.com_google_containers_pause_3.2.tar，三个节点都导入。
配置文件说明：
--hostname-override：显示名称，集群中唯一 --network-plugin：启用CNI --kubeconfig：空路径，会自动生成，后面用于连接apiserver --bootstrap-kubeconfig：首次启动向apiserver申请证书 --config：配置参数文件 --cert-dir：kubelet证书生成目录 --pod-infra-container-image：管理Pod网络容器的镜像
这里要注意一个难点，-kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig这一段，表示这个文件会在服务启动的时候自动生成，但一般稍微有点错，它就生成不了，比如下面的文件如果有写错，那么，将不会自动生成这个文件。
vim /opt/kubernetes/cfg/kubelet-config.yml
kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 0.0.0.0 port: 10250 readOnlyPort: 10255 cgroupDriver: cgroupfs clusterDNS: - 10.0.0.2 clusterDomain: cluster.local failSwapOn: false authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /opt/kubernetes/ssl/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5% maxOpenFiles: 1000000 maxPods: 110
生成bootstrap.kubeconfig文件：
1. KUBE_APISERVER="https://192.168.217.16:6443" 2. TOKEN="c47ffb939f5ca36231d9e3121a252940"
