1.6、部署kube-apiserver
- 所有
master节点需要kube-apiserver kube-apiserver是无状态服务，需要通过kube-nginx进行代理访问，从而保证服务可用性
- 部署kubectl的时候已经下载了完整的kubernetes二进制文件，因此kube-apiserver就无须下载了，等下脚本分发即可
1.6.0、创建kubernetes证书和私钥
k8s-01:~ # cd /opt/k8s/ssl/ k8s-01:/opt/k8s/ssl # source /opt/k8s/bin/k8s-env.sh k8s-01:/opt/k8s/ssl # cat > kubernetes-csr.json <<EOF { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.72.39", "192.168.72.40", "192.168.72.41", "${CLUSTER_KUBERNETES_SVC_IP}", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "k8s", "OU": "bandian" } ] } EOF
- 需要将
集群的所有IP添加到证书内
1.6.1、生成kubernetes证书和私钥
k8s-01:/opt/k8s/ssl # cfssl gencert -ca=/opt/k8s/ssl/ca.pem \ -ca-key=/opt/k8s/ssl/ca-key.pem \ -config=/opt/k8s/ssl/ca-config.json \ -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
1.6.2、创建metrics-server证书和私钥
k8s-01:/opt/k8s/ssl # cat > metrics-server-csr.json <<EOF { "CN": "aggregator", "hosts": [ ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "k8s", "OU": "bandian" } ] } EOF
1.6.3、生成metrics-server证书和私钥
k8s-01:/opt/k8s/ssl # cfssl gencert -ca=/opt/k8s/ssl/ca.pem \ -ca-key=/opt/k8s/ssl/ca-key.pem \ -config=/opt/k8s/ssl/ca-config.json \ -profile=kubernetes metrics-server-csr.json | cfssljson -bare metrics-server
1.6.4、配置kube-apiserver为systemctl管理
k8s-01:~ # cd /opt/k8s/conf/ k8s-01:/opt/k8s/conf # source /opt/k8s/bin/k8s-env.sh k8s-01:/opt/k8s/conf # cat > kube-apiserver.service.template <<EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] WorkingDirectory=${K8S_DIR}/kube-apiserver ExecStart=/opt/k8s/bin/kube-apiserver \\ --v=2 \\ --advertise-address=##NODE_IP## \\ --secure-port=6443 \\ --bind-address=##NODE_IP## \\ --etcd-servers=${ETCD_ENDPOINTS} \\ --allow-privileged=true \\ --service-cluster-ip-range=${SERVICE_CIDR} \\ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\ --authorization-mode=RBAC,Node \\ --enable-bootstrap-token-auth=true \\ --token-auth-file=/etc/kubernetes/cert/token.csv \\ --service-node-port-range=${NODE_PORT_RANGE} \\ --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\ --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\ --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\ --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\ --client-ca-file=/etc/kubernetes/cert/ca.pem \\ --service-account-key-file=/etc/kubernetes/cert/ca.pem \\ --etcd-cafile=/etc/kubernetes/cert/ca.pem \\ --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\ --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\ --audit-log-maxage=15 \\ --audit-log-maxbackup=3 \\ --audit-log-maxsize=100 \\ --audit-log-truncate-enabled \\ --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\ --proxy-client-cert-file=/etc/kubernetes/cert/metrics-server.pem \\ --proxy-client-key-file=/etc/kubernetes/cert/metrics-server-key.pem \\ --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\ --requestheader-allowed-names=aggregator \\ --requestheader-extra-headers-prefix="X-Remote-Extra-" \\ --requestheader-group-headers=X-Remote-Group \\ --requestheader-username-headers=X-Remote-User Restart=on-failure RestartSec=10 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
--v日志等级--etcd-serversetcd集群地址--bind-address监听地址--secure-porthttps安全端口--advertise-address集群通告地址--allow-privileged启用授权--service-cluster-ip-rangeService虚拟IP地址段--enable-admission-plugins准入控制模块--authorization-mode认证授权，启用RBAC授权和节点自管理--enable-bootstrap-token-auth启用TLS bootstrap机制--token-auth-filebootstrap token文件--service-node-port-rangeService nodeport类型默认分配端口范围--kubelet-client-xxxapiserver访问kubelet客户端证书--tls-xxx-fileapiserver https证书--etcd-xxxfile连接Etcd集群证书 --audit-log-xxx:审计日志
--requestheader-xxx-xxx开启kube-apiserver的aggregation（hpa和metrics依赖aggregation）--proxy-client-xxx同上
1.6.5、配置bootstrap token文件
k8s-01:~ # cd /opt/k8s/ssl/ k8s-01:/opt/k8s/ssl # cat > token.csv <<EOF 404a083c42f5d39979fd731a24774b83,kubelet-bootstrap,10001,"system:node-bootstrapper" EOF
- bootstrap token文件格式
token，用户名，UID，用户组
- token生成方式
head -c 16 /dev/urandom | od -An -t x | tr -d ' '
1.6.6、分发kube-apiserver命令和秘钥等文件到其他节点
#!/usr/bin/env bash source /opt/k8s/bin/k8s-env.sh # 替换模板文件 for (( i=0; i < 3; i++ )) do sed -e "s/##NODE_IP##/${MASTER_IPS[i]}/" /opt/k8s/conf/kube-apiserver.service.template > \ /opt/k8s/conf/kube-apiserver-${MASTER_IPS[i]}.service done # 分发到master节点 for host in ${MASTER_IPS[@]} do printf "\e[1;34m${host}\e[0m\n" scp /opt/k8s/packages/kubernetes/server/bin/{apiextensions-apiserver,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubelet,mounter} ${host}:/opt/k8s/bin/ scp /opt/k8s/ssl/{kubernetes*.pem,token.csv} ${host}:/etc/kubernetes/cert/ scp /opt/k8s/ssl/metrics-server*.pem ${host}:/etc/kubernetes/cert/ scp /opt/k8s/conf/kube-apiserver-${host}.service ${host}:/etc/systemd/system/kube-apiserver.service done # 分发到所有节点 for host_node in ${NODE_IPS[@]} do printf "\e[1;34m${host_node}\e[0m\n" scp /opt/k8s/packages/kubernetes/server/bin/{kubelet,kube-proxy} ${host_node}:/opt/k8s/bin/ done
1.6.7、启动kube-apiserver服务
#!/usr/bin/env bash source /opt/k8s/bin/k8s-env.sh for host in ${MASTER_IPS[@]} do printf "\e[1;34m${host}\e[0m\n" ssh root@${host} "mkdir -p ${K8S_DIR}/kube-apiserver/" ssh root@${host} "systemctl daemon-reload && \ systemctl enable kube-apiserver --now && \ systemctl status kube-apiserver | grep Active" done
注：返回的如果是Active: activating (auto-restart)，可以稍等一下，然后再次执行systemctl status kube-apiserver | grep Active，
- 出现running就可以了，否则的话，需要查看日志
journalctl -xeu kube-apiserver
1.6.8、查看kube-apiserver写入etcd的数据
k8s-01:~ # source /opt/k8s/bin/k8s-env.sh k8s-01:~ # etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --cacert=/opt/k8s/ssl/ca.pem \ --cert=/opt/k8s/ssl/etcd.pem \ --key=/opt/k8s/ssl/etcd-key.pem \ get /registry/ --prefix --keys-only
1.6.9、检查kubernetes集群信息
k8s-01:~ # kubectl cluster-info Kubernetes master is running at https://192.168.72.39:8443 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. k8s-01:~ # kubectl get all --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 38s k8s-01:~ # kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused controller-manager Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused etcd-1 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"} etcd-0 Healthy {"health":"true"}
- 注：如果有报错，检查一下
~/.kube/config的配置，以及证书是否正确
1.6.10、授权kubelet-bootstrap用户允许请求证书
k8s-01:~ # kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes
