kubeadm是Kubernetes官方提供的快速安装和初始化Kubernetes集群的工具，目前的还处于孵化开发状态，伴随Kubernetes每个版本的发布都会同步更新。 当然，目前的kubeadm是不能用于生产环境的。 但伴随着Kubernetes每次版本升级，kubeadm都会对集群配置方面的一些实践做调整，通过实验kubeadm我们可以学习到Kubernetes官方在集群配置上一些新的最佳实践。
Kubernetes 1.8已经发布，为了跟上官方更新的脚本，接下来体验一下Kubernetes 1.8中的kubeadm。
1.准备
1.1系统配置
在安装之前，需要先做如下准备。两台CentOS 7.3主机如下：
cat /etc/hosts 192.168.61.11 node1 192.168.61.12 node2
如果各个主机启用了防火墙，需要开放Kubernetes各个组件所需要的端口，可以查看Installing kubeadm中的”Check required ports”一节。 这里简单起见在各节点禁用防火墙：
systemctl stop firewalld
systemctl disable firewalld
创建/etc/sysctl.d/k8s.conf文件，添加如下内容：
net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1
执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。
禁用SELINUX：
setenforce 0
vi /etc/selinux/config SELINUX=disabled
Kubernetes 1.8开始要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动。可以通过kubelet的启动参数–fail-swap-on=false更改这个限制。 我们这里关闭系统的Swap:
swapoff -a
修改 /etc/fstab 文件，注释掉 SWAP 的自动挂载，使用free -m确认swap已经关闭。
swappiness参数调整，修改/etc/sysctl.d/k8s.conf添加下面一行：
vm.swappiness=0
执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。
1.2安装Docker
yum install -y yum-utils device-mapper-persistent-data lvm2 yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo
查看当前的Docker版本：
yum list docker-ce.x86_64 --showduplicates |sort -r docker-ce.x86_64 17.09.0.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.06.2.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.06.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.06.0.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.03.2.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.03.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.03.0.ce-1.el7.centos docker-ce-stable
Kubernetes 1.8已经针对Docker的1.11.2, 1.12.6, 1.13.1和17.03.2等版本做了验证。 因为我们这里在各节点安装docker的17.03.2版本。
yum makecache fast yum install -y --setopt=obsoletes=0 \ docker-ce-17.03.2.ce-1.el7.centos \ docker-ce-selinux-17.03.2.ce-1.el7.centos systemctl start docker systemctl enable docker
Docker从1.13版本开始调整了默认的防火墙规则，禁用了iptables filter表中FOWARD链，这样会引起Kubernetes集群中跨Node的Pod无法通信，在各个Docker节点执行下面的命令：
iptables -P FORWARD ACCEPT
可在docker的systemd unit文件中以ExecStartPost加入上面的命令：
ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT
systemctl daemon-reload systemctl restart docker
2.安装kubeadm和kubelet
下面在各节点安装kubeadm和kubelet：
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF
测试地址https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64是否可用，如果不可用需要科学上网。
curl https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
yum makecache fast yum install -y kubelet kubeadm kubectl ... Installed: kubeadm.x86_64 0:1.8.0-0 kubectl.x86_64 0:1.8.0-0 kubelet.x86_64 0:1.8.0-0 Dependency Installed: kubernetes-cni.x86_64 0:0.5.1-0 socat.x86_64 0:1.7.3.2-2.el7
- 从安装结果可以看出还安装了kubernetes-cni和socat两个依赖：
- 可以看出官方Kubernetes 1.8依赖的cni还是0.5.1版本
- socat是kubelet的依赖
- 我们之前在Kubernetes 1.6 高可用集群部署中手动安装这两个依赖的
Kubernetes文档中kubelet的启动参数：
--cgroup-driver string Driver that the kubelet uses to manipulate cgroups on the host. Possible values: 'cgroupfs', 'systemd' (default "cgroupfs")
默认值为cgroupfs，但是我们注意到yum安装kubelet,kubeadm时生成10-kubeadm.conf文件中将这个参数值改成了systemd。
查看kubelet的 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf文件，其中包含如下内容：
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
使用docker info打印docker信息：
docker info ...... Server Version: 17.03.2-ce ...... Cgroup Driver: cgroupfs
可以看出docker 17.03使用的Cgroup Driver为cgroupfs。
于是修改各节点docker的cgroup driver使其和kubelet一致，即修改或创建/etc/docker/daemon.json，加入下面的内容：
{ "exec-opts": ["native.cgroupdriver=systemd"] }
重启docker:
systemctl restart docker
systemctl status docker
在各节点开机启动kubelet服务：
systemctl enable kubelet.service
3.使用kubeadm init初始化集群
接下来使用kubeadm初始化集群，选择node1作为Master Node，在node1上执行下面的命令：
kubeadm init \ --kubernetes-version=v1.8.0 \ --pod-network-cidr=10.244.0.0/16 \ --apiserver-advertise-address=192.168.61.11
因为我们选择flannel作为Pod网络插件，所以上面的命令指定–pod-network-cidr=10.244.0.0/16。
kubeadm init \ > --kubernetes-version=v1.8.0 \ > --pod-network-cidr=10.244.0.0/16 \ > --apiserver-advertise-address=192.168.61.11 [kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters. [init] Using Kubernetes version: v1.8.0 [init] Using Authorization modes: [Node RBAC] [preflight] Running pre-flight checks [preflight] WARNING: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly [preflight] Starting the kubelet service [kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0) [certificates] Generated ca certificate and key. [certificates] Generated apiserver certificate and key. [certificates] apiserver serving cert is signed for DNS names [node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.61.11] [certificates] Generated apiserver-kubelet-client certificate and key. [certificates] Generated sa key and public key. [certificates] Generated front-proxy-ca certificate and key. [certificates] Generated front-proxy-client certificate and key. [certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki" [kubeconfig] Wrote KubeConfig file to disk: "admin.conf" [kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf" [kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf" [kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf" [controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml" [controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml" [controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml" [etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml" [init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests" [init] This often takes around a minute; or longer if the control plane images have to be pulled. [apiclient] All control plane components are healthy after 28.505733 seconds [uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [markmaster] Will mark node node1 as master by adding a label and a taint [markmaster] Master node1 tainted and labelled with key/value: node-role.kubernetes.io/master="" [bootstraptoken] Using token: 9e68dd.7117f03c900f9234 [bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: kube-dns [addons] Applied essential addon: kube-proxy Your Kubernetes master has initialized successfully! To start using your cluster, you need to run (as a regular user): mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: http://kubernetes.io/docs/admin/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join --token 9e68dd.7117f03c900f9234 192.168.61.11:6443 --discovery-token-ca-cert-hash sha256:82a08ef9c830f240e588a26a8ff0a311e6fe3127c1ee4c5fc019f1369007c0b7
上面记录了完成的初始化输出的内容。
其中由以下关键内容：
- kubeadm 1.8当前还处于beta状态，还不能用于生产环境。目前来看这东西安装的etcd和apiserver都是单节点，当然不能用于生产环境。
- RBAC模式已经在Kubernetes 1.8中稳定可用。kubeadm 1.8也默认启用了RBAC
- 接下来是生成证书和相关的kubeconfig文件，这个目前我们在Kubernetes 1.6 高可用集群部署也是这么做的，目前没看出有什么新东西
- 生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
- 另外注意kubeadm还报了starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use –token-ttl 0)的警告
- 下面的命令是配置常规用户如何使用kubectl访问集群：
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config- 最后给出了将节点加入集群的命令kubeadm join –token 9e68dd.7117f03c900f9234 192.168.61.11:6443 –discovery-token-ca-cert-hash sha256:82a08ef9c830f240e588a26a8ff0a311e6fe3127c1ee4c5fc019f1369007c0b7
查看一下集群状态：
kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health": "true"}
确认个组件都处于healthy状态。
集群初始化如果遇到问题，可以使用下面的命令进行清理：
kubeadm reset ifconfig cni0 down ip link delete cni0 ifconfig flannel.1 down ip link delete flannel.1 rm -rf /var/lib/cni/
4.安装Pod Network
接下来安装flannel network add-on：
mkdir -p ~/k8s/ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml kubectl apply -f kube-flannel.yml clusterrole "flannel" created clusterrolebinding "flannel" created serviceaccount "flannel" created configmap "kube-flannel-cfg" created daemonset "kube-flannel-ds" created
这里注意kube-flannel.yml这个文件中已经包含了ServiceAccount, ClusterRole和ClusterRoleBinding，原来是在一个单独的kube-flannel-rbac.yml文件中。kube-flannel.yml这个文件里的flannel的镜像是0.9.0，quay.io/coreos/flannel:v0.9.0-amd64
如果Node有多个网卡的话，参考flannel issues 39701，目前需要在kube-flannel.yml中使用–iface参数指定集群主机内网网卡的名称，否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地，flanneld启动参数加上–iface=<iface-name>
...... apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: kube-flannel-ds ...... containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.9.0-amd64 command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=eth1" ] ......
使用kubectl get pod –all-namespaces -o wide确保所有的Pod都处于Running状态。
kubectl get pod --all-namespaces -o wide
5.master node参与工作负载
使用kubeadm初始化的集群，出于安全考虑Pod不会被调度到Master Node上，也就是说Master Node不参与工作负载。
这里搭建的是测试环境可以使用下面的命令使Master Node参与工作负载：
kubectl taint nodes node1 node-role.kubernetes.io/master- node "node1" untainted
6.测试DNS
kubectl run curl --image=radial/busyboxplus:curl -i --tty If you don't see a command prompt, try pressing enter. [ root@curl-2716574283-xr8zd:/ ]$
进入后执行nslookup kubernetes.default确认解析正常:
nslookup kubernetes.default Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
7.向Kubernetes集群添加Node
下面我们将k8s-node2这个主机添加到Kubernetes集群中，在k8s-node2上执行：
kubeadm join --token 9e68dd.7117f03c900f9234 192.168.61.11:6443 --discovery-token-ca-cert-hash sha256:82a08ef9c830f240e588a26a8ff0a311e6fe3127c1ee4c5fc019f1369007c0b7 [kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters. [preflight] Running pre-flight checks [discovery] Trying to connect to API Server "192.168.61.11:6443" [discovery] Created cluster-info discovery client, requesting info from "https://192.168.61.11:6443" [discovery] Requesting info from "https://192.168.61.11:6443" again to validate TLS against the pinned public key [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.61.11:6443" [discovery] Successfully established connection with API Server "192.168.61.11:6443" [bootstrap] Detected server version: v1.8.0 [bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1) Node join complete: * Certificate signing request sent to master and response received. * Kubelet informed of new secure connection details. Run 'kubectl get nodes' on the master to see this machine join.
这次很是顺利，下面在master节点上执行命令查看集群中的节点：
kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready master 25m v1.8.0 node2 Ready <none> 10m v1.8.0
如何从集群中移除Node
如果需要从集群中移除node2这个Node执行下面的命令：
在master节点上执行：
kubectl drain node2 --delete-local-data --force --ignore-daemonsets kubectl delete node node2
在node2上执行：
kubeadm reset ifconfig cni0 down ip link delete cni0 ifconfig flannel.1 down ip link delete flannel.1 rm -rf /var/lib/cni/
8.dashboard插件部署
注意当前dashboard的版本已经是1.7.1了。 而1.7.x版本的dashboard对安全做了增强，默认需要以https的方式访问，增加了登录的页面，同时增加了一个gcr.io/google_containers/kubernetes-dashboard-init-amd64的init容器。
另外需要注意dashboard调整了部署文件的源码目录结构：
mkdir -p ~/k8s/ wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml kubectl create -f kubernetes-dashboard.yaml
kubernetes-dashboard.yaml文件中的ServiceAccount kubernetes-dashboard只有相对较小的权限，因此我们创建一个kubernetes-dashboard-admin的ServiceAccount并授予集群admin的权限，创建kubernetes-dashboard-admin.rbac.yaml：
--- apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-admin namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard-admin labels: k8s-app: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: kubernetes-dashboard-admin namespace: kube-system
kubectl create -f kubernetes-dashboard-admin.rbac.yaml serviceaccount "kubernetes-dashboard-admin" created clusterrolebinding "kubernetes-dashboard-admin" created
查看kubernete-dashboard-admin的token:
kubectl -n kube-system get secret | grep kubernetes-dashboard-admin kubernetes-dashboard-admin-token-pfss5 kubernetes.io/service-account-token 3 14s kubectl describe -n kube-system secret/kubernetes-dashboard-admin-token-pfss5 Name: kubernetes-dashboard-admin-token-pfss5 Namespace: kube-system Labels: <none> Annotations: kubernetes.io/service-account.name=kubernetes-dashboard-admin kubernetes.io/service-account.uid=1029250a-ad76-11e7-9a1d-08002778b8a1 Type: kubernetes.io/service-account-token Data ==== ca.crt: 1025 bytes namespace: 11 bytes token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi1wZnNzNSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjEwMjkyNTBhLWFkNzYtMTFlNy05YTFkLTA4MDAyNzc4YjhhMSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.Bs6h65aFCFkEKBO_h4muoIK3XdTcfik-pNM351VogBJD_pk5grM1PEWdsCXpR45r8zUOTpGM-h8kDwgOXwy2i8a5RjbUTzD3OQbPJXqa1wBk0ABkmqTuw-3PWMRg_Du8zuFEPdKDFQyWxiYhUi_v638G-R5RdZD_xeJAXmKyPkB3VsqWVegoIVTaNboYkw6cgvMa-4b7IjoN9T1fFlWCTZI8BFXbM8ICOoYMsOIJr3tVFf7d6oVNGYqaCk42QL_2TfB6xMKLYER9XDh753-_FDVE5ENtY5YagD3T_s44o0Ewara4P9C3hYRKdJNLxv7qDbwPl3bVFH3HXbsSxxF3TQ
在dashboard的登录窗口使用上面的token登录。
9.heapster插件部署
下面安装Heapster为集群添加使用统计和监控功能，为Dashboard添加仪表盘。 使用InfluxDB做为Heapster的后端存储，开始部署：
mkdir -p ~/k8s/heapster cd ~/k8s/heapster wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml kubectl create -f ./
最后确认所有的pod都处于running状态，打开Dashboard,集群的使用统计会以仪表盘的形式显示出来。
本次安装涉及到的Docker镜像：
gcr.io/google_containers/kube-apiserver-amd64:v1.8.0 gcr.io/google_containers/kube-controller-manager-amd64:v1.8.0 gcr.io/google_containers/kube-scheduler-amd64:v1.8.0 gcr.io/google_containers/kube-proxy-amd64:v1.8.0 gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5 gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5 gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5 quay.io/coreos/flannel:v0.9.0-amd64 gcr.io/google_containers/etcd-amd64:3.0.17 gcr.io/google_containers/pause-amd64:3.0 quay.io/coreos/flannel:v0.9.0-amd64 gcr.io/google_containers/kubernetes-dashboard-amd64:v1.7.1 gcr.io/google_containers/kubernetes-dashboard-init-amd64:v1.0.0 gcr.io/google_containers/heapster-influxdb-amd64:v1.3.3 gcr.io/google_containers/heapster-grafana-amd64:v4.4.3 gcr.io/google_containers/heapster-amd64:v1.4.0
