K8S高可用安装手册

K8S-kubeadm安装高可用（完成版本）

1 主机列表

共有5台设备，3台master，1台work

2 K8S版本

3 高可用架构

采用kubeadm来构建k8s集群，apiserver利用vip进行构建，kubelet连接vip，实现高可用的目的

4 高可用说明

• apiserver 通过keepalived实现高可用，当某个节点故障时触发keepalived vip 转移；
• controller-manager k8s内部通过选举方式产生领导者(由--leader-elect 选型控制，默认为true)，同一时刻集群内只有一个controller-manager组件运行；
• scheduler k8s内部通过选举方式产生领导者(由--leader-elect 选型控制，默认为true)，同一时刻集群内只有一个scheduler组件运行；
• etcd 通过运行kubeadm方式自动创建集群来实现高可用，部署的节点数为奇数，3节点方式最多容忍一台机器宕机。

5 安装准备工作

所有设备master01,master02,master03,work01,work02上执行

5.1  修改主机名

hostnamectl set-hostname master01

hostnamectl set-hostname master02

hostnamectl set-hostname master03

hostnamectl set-hostname work01

hostnamectl set-hostname work02

systemctl restart systemd-hostnamed

5.2 修改host文件

cat >> /etc/hosts << EOF

192.168.59.11 master01

192.168.59.12 master02

192.168.59.13 master03

192.168.59.21 work01

192.168.59.22 work02

EOF

5.3 修改DNS

cat >> /etc/resolv.conf << EOF

nameserver 114.114.114.114

nameserver 8.8.8.8

EOF

5.4 关闭SELINUX

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

5.5 关闭防火墙

#查看防火墙状态

systemctl status firewalld.service

#状态是运行中，需要关闭

Active: active (running)

#关闭防火墙

systemctl stop firewalld

#再次查看状态确认

#设置开机关闭，不要启动

systemctl disable firewalld

5.6 设置iptables为空规则

yum -y install iptables-services

#安装iptablesservices

systemctl start iptables

#启动防火墙

systemctl enable iptables

#设置防火墙卡开机启动

iptables -F

service iptables save

#清空规则，并保存

5.7 关闭swap分区

sed -i.bak '/swap/s/^/#/' /etc/fstab

swapoff -a

5.8 加载模块

k8s网络使用的flannel，该网络需要使用内核的br_netfilter模块

#查询是否加载了这个模块

lsmod |grep br_netfilter

#永久加载模块

cat > /etc/rc.sysinit << EOF

#!/bin/bash

for file in /etc/sysconfig/modules/*.modules ; do

[ -x $file ] && $file

done

EOF

cat > /etc/sysconfig/modules/br_netfilter.modules << EOF

modprobe br_netfilter

EOF

chmod 755 /etc/sysconfig/modules/br_netfilter.modules

5.9 设置内核参数

cat << EOF > /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

vm.swappiness=0

net.ipv4.ip_forward=1

net.ipv4.tcp_tw_recycle=0

vm.overcommit_memory=1

vm.panic_on_oom=0

fs.inotify.max_user_instances=8192

fs.inotify.max_user_watches=1048576

fs.file-max=52706963

fs.nr_open=52706963

net.ipv6.conf.all.disable_ipv6=1

net.netfileter.nf_conntrack_max=2310720

EOF

5.10 安装常用软件包

yum -y install epel-release vim net-tools gcc gcc-c++ glibc htop atop iftop iotop nethogs lrzsz telnet ipvsadm ipset conntrack  libnl-devel  libnfnetlink-devel openssl openssl-devel contrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git

5.11 同步时间

yum install ntpdate -y &&/usr/sbin/ntpdate -u ntp1.aliyun.com

5.12 关闭不需要服务

systemctl stop postfix && systemctl disable postfix

5.13 配置k8s YUM源

5.13.1新增阿里云yum

cat << EOF > /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/

enabled=1

gpgcheck=1

repo_gpgcheck=1

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

EOF

• [] 中括号中的是repository id，唯一，用来标识不同仓库
• name 仓库名称，自定义
• baseurl 仓库地址
• enable 是否启用该仓库，默认为1表示启用
• gpgcheck 是否验证从该仓库获得程序包的合法性，1为验证
• repo_gpgcheck 是否验证元数据的合法性 元数据就是程序包列表，1为验证
• gpgkey=URL 数字签名的公钥文件所在位置，如果gpgcheck值为1，此处就需要指定gpgkey文件的位置，如果gpgcheck值为0就不需要此项了

5.13.2 更新缓存

yum clean all && yum -y makecache

5.14 SSH无密码认证

这里在master01上配置无密码认证，方便连接master02,master03上

[root@master01 ~]#ssh-keygen -t rsa

[root@master01 ~]#ssh-copy-id -i /root/.ssh/id_rsa.pub root@master02

[root@master01 ~]#ssh-copy-id -i /root/.ssh/id_rsa.pub root@master03

6 Docker安装

master01,master02,master03,work01,work02上执行安装

6.1 安装依赖包

yum install -y yum-utils   device-mapper-persistent-data  lvm2

6.2 设置docker源

yum-config-manager --add-repo     https://download.docker.com/linux/centos/docker-ce.repo

6.3 查看docker ce版本

yum list docker-ce --showduplicates | sort -r

Repository epel is listed more than once in the configuration

Repository epel-debuginfo is listed more than once in the configuration

Repository epel-source is listed more than once in the configuration

已加载插件：fastestmirror

可安装的软件包

* updates: mirrors.aliyun.com

Loading mirror speeds from cached hostfile

* extras: mirrors.aliyun.com

docker-ce.x86_64            3:19.03.9-3.el7                     docker-ce-stable

docker-ce.x86_64            3:19.03.8-3.el7                     docker-ce-stable

docker-ce.x86_64            3:19.03.7-3.el7                     docker-ce-stable

docker-ce.x86_64            3:19.03.6-3.el7                     docker-ce-stable

docker-ce.x86_64            3:19.03.5-3.el7                     docker-ce-stable

docker-ce.x86_64            3:19.03.4-3.el7                     docker-ce-stable

docker-ce.x86_64            3:19.03.3-3.el7                     docker-ce-stable

docker-ce.x86_64            3:19.03.2-3.el7                     docker-ce-stable

docker-ce.x86_64            3:19.03.1-3.el7                     docker-ce-stable

docker-ce.x86_64            3:19.03.13-3.el7                    docker-ce-stable

docker-ce.x86_64            3:19.03.12-3.el7                    docker-ce-stable

docker-ce.x86_64            3:19.03.11-3.el7                    docker-ce-stable

docker-ce.x86_64            3:19.03.10-3.el7                    docker-ce-stable

docker-ce.x86_64            3:19.03.0-3.el7                     docker-ce-stable

docker-ce.x86_64            3:18.09.9-3.el7                     docker-ce-stable

docker-ce.x86_64            3:18.09.8-3.el7                     docker-ce-stable

docker-ce.x86_64            3:18.09.7-3.el7                     docker-ce-stable

docker-ce.x86_64            3:18.09.6-3.el7                     docker-ce-stable

docker-ce.x86_64            3:18.09.5-3.el7                     docker-ce-stable

docker-ce.x86_64            3:18.09.4-3.el7                     docker-ce-stable

docker-ce.x86_64            3:18.09.3-3.el7                     docker-ce-stable

docker-ce.x86_64            3:18.09.2-3.el7                     docker-ce-stable

docker-ce.x86_64            3:18.09.1-3.el7                     docker-ce-stable

docker-ce.x86_64            3:18.09.0-3.el7                     docker-ce-stable

docker-ce.x86_64            18.06.3.ce-3.el7                    docker-ce-stable

docker-ce.x86_64            18.06.2.ce-3.el7                    docker-ce-stable

docker-ce.x86_64            18.06.1.ce-3.el7                    docker-ce-stable

docker-ce.x86_64            18.06.0.ce-3.el7                    docker-ce-stable

docker-ce.x86_64            18.03.1.ce-1.el7.centos             docker-ce-stable

docker-ce.x86_64            18.03.0.ce-1.el7.centos             docker-ce-stable

docker-ce.x86_64            17.12.1.ce-1.el7.centos             docker-ce-stable

docker-ce.x86_64            17.12.0.ce-1.el7.centos             docker-ce-stable

docker-ce.x86_64            17.09.1.ce-1.el7.centos             docker-ce-stable

docker-ce.x86_64            17.09.0.ce-1.el7.centos             docker-ce-stable

docker-ce.x86_64            17.06.2.ce-1.el7.centos             docker-ce-stable

docker-ce.x86_64            17.06.1.ce-1.el7.centos             docker-ce-stable

docker-ce.x86_64            17.06.0.ce-1.el7.centos             docker-ce-stable

docker-ce.x86_64            17.03.3.ce-1.el7                    docker-ce-stable

docker-ce.x86_64            17.03.2.ce-1.el7.centos             docker-ce-stable

docker-ce.x86_64            17.03.1.ce-1.el7.centos             docker-ce-stable

docker-ce.x86_64            17.03.0.ce-1.el7.centos             docker-ce-stable

* base: mirrors.aliyun.com

6.4 安装指定版本docker

yum install docker-ce-18.09.9 docker-ce-cli-18.09.9 containerd.io -y

6.5 启动docker

systemctl start docker

systemctl enable docker

7 安装命令补全

所有设备安装

yum -y install bash-completion

source /etc/profile.d/bash_completion.sh

8 镜像加速

所有设备安装

默认的docker下载地址速度很慢，这里用阿里云的镜像加速来作为镜像站

登陆地址为：https://cr.console.aliyun.com ,未注册的可以先注册阿里云账户

8.1 配置加速

mkdir -p /etc/docker

cat > /etc/docker/daemon.json <<EOF

{

 "registry-mirrors": ["https://lss3ndia.mirror.aliyuncs.com"],

 "exec-opts": ["native.cgroupdriver=systemd"],

 "log-driver": "json-file",

 "log-opts": {

   "max-size": "100m"

 },

 "storage-driver": "overlay2"

}

EOF

8.2 重启docker

systemctl daemon-reload && systemctl restart docker

8.3 验证

docker --version

Docker version 18.09.9, build 039a7df9ba

9 Keepalived安装

在mster01,master02,master03上安装

yum -y install keepalived

9.1 配置keepalived

master01上

[root@master01 ~]#more /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

  router_id master01

}

vrrp_instance VI_1 {

   state MASTER

   interface ens32

   virtual_router_id 50

   priority 100

   advert_int 1

   authentication {

       auth_type PASS

       auth_pass 1111

   }

   virtual_ipaddress {

       192.168.59.10

   }

}

master02上

[root@master02 ~]#more /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

  router_id master02

}

vrrp_instance VI_1 {

   state BACKUP

   interface ens32

   virtual_router_id 50

   priority 90

   advert_int 1

   authentication {

       auth_type PASS

       auth_pass 1111

   }

   virtual_ipaddress {

       192.168.59.10

   }

}

master03上

[root@master03 ~]#more /etc/keepalived/keepalived.conf    

! Configuration File for keepalived

global_defs {

  router_id master03

}

vrrp_instance VI_1 {

   state BACKUP

   interface ens32

   virtual_router_id 50

   priority 80

   advert_int 1

   authentication {

       auth_type PASS

       auth_pass 1111

   }

   virtual_ipaddress {

       192.168.59.10

   }

9.2 启动服务

service keepalived start

systemctl enable keepalived

10 安装K8S

所有节点安装

10.1 查看版本

本文安装的kubelet版本是1.16.4，该版本支持的docker版本为1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09

yum list kubelet --showduplicates | sort -r

10.2 安装指定版本

这里安装16.4版本

yum install -y kubelet-1.16.4 kubeadm-1.16.4 kubectl-1.16.4

10.3 安装包说明

kubelet 运行在集群所有节点上，用于启动Pod和容器等对象的工具

kubeadm 用于初始化集群，启动集群的命令工具

kubectl 用于和集群通信的命令行，通过kubectl可以部署和管理应用，查看各种资源，创建、删除和更新各种组件

10.4 启动kubelet

systemctl enable kubelet && systemctl start kubelet

10.5 kubectl命令补全

echo "source <(kubectl completion bash)" >> ~/.bash_profile

source .bash_profile

11 镜像包下载

因为默认的k8s的镜像都在google，访问不了，这里用互联网上的分享的源下载

10月28号补充，我已经将所有的镜像打包成k8s_1.16.4.tar.gz，使用时候导入镜像即可

所有master，所有work上

docker pull registry.cn-hangzhou.aliyuncs.com/loong576/kube-apiserver:v1.16.4

docker pull registry.cn-hangzhou.aliyuncs.com/loong576/kube-controller-manager:v1.16.4

docker pull registry.cn-hangzhou.aliyuncs.com/loong576/kube-scheduler:v1.16.4

docker pull registry.cn-hangzhou.aliyuncs.com/loong576/kube-proxy:v1.16.4

docker pull registry.cn-hangzhou.aliyuncs.com/loong576/pause:3.1

docker pull registry.cn-hangzhou.aliyuncs.com/loong576/etcd:3.3.15-0

docker pull registry.cn-hangzhou.aliyuncs.com/loong576/coredns:1.6.2

11.1 查看下载的镜像

[root@master01 ~]#docker images

REPOSITORY                                                           TAG                 IMAGE ID            CREATED             SIZE

hello-world                                                          latest              bf756fb1ae65        9 months ago        13.3kB

registry.cn-hangzhou.aliyuncs.com/loong576/kube-apiserver            v1.16.4             3722a80984a0        10 months ago       217MB

registry.cn-hangzhou.aliyuncs.com/loong576/kube-controller-manager   v1.16.4             fb4cca6b4e4c        10 months ago       163MB

registry.cn-hangzhou.aliyuncs.com/loong576/kube-scheduler            v1.16.4             2984964036c8        10 months ago       87.3MB

registry.cn-hangzhou.aliyuncs.com/loong576/kube-proxy                v1.16.4             091df896d78f        10 months ago       86.1MB

registry.cn-hangzhou.aliyuncs.com/loong576/etcd                      3.3.15-0            b2756210eeab        13 months ago       247MB

registry.cn-hangzhou.aliyuncs.com/loong576/coredns                   1.6.2               bf261d157914        14 months ago       44.1MB

registry.cn-hangzhou.aliyuncs.com/loong576/pause                     3.1                 da86e6ba6ca1        2 years ago         742kB

11.2 镜像改名字

因为k8s adm工具初始化的时候，查找有没有这些镜像，而且是按照名字来找的，不修改名字，还是从默认的google来下载

docker tag registry.cn-hangzhou.aliyuncs.com/loong576/kube-apiserver:v1.16.4 k8s.gcr.io/kube-apiserver:v1.16.4

docker tag registry.cn-hangzhou.aliyuncs.com/loong576/kube-controller-manager:v1.16.4 k8s.gcr.io/kube-controller-manager:v1.16.4

docker tag registry.cn-hangzhou.aliyuncs.com/loong576/kube-scheduler:v1.16.4 k8s.gcr.io/kube-scheduler:v1.16.4

docker tag registry.cn-hangzhou.aliyuncs.com/loong576/kube-proxy:v1.16.4 k8s.gcr.io/kube-proxy:v1.16.4

docker tag registry.cn-hangzhou.aliyuncs.com/loong576/etcd:3.3.15-0 k8s.gcr.io/etcd:3.3.15-0

docker tag registry.cn-hangzhou.aliyuncs.com/loong576/coredns:1.6.2 k8s.gcr.io/coredns:1.6.2

docker tag registry.cn-hangzhou.aliyuncs.com/loong576/pause:3.1 k8s.gcr.io/pause:3.1

#确认有如下镜像

[root@master01 ~]#docker images | grep k8s

k8s.gcr.io/kube-apiserver                                            v1.16.4             3722a80984a0        10 months ago       217MB

k8s.gcr.io/kube-controller-manager                                   v1.16.4             fb4cca6b4e4c        10 months ago       163MB

k8s.gcr.io/kube-scheduler                                            v1.16.4             2984964036c8        10 months ago       87.3MB

k8s.gcr.io/kube-proxy                                                v1.16.4             091df896d78f        10 months ago       86.1MB

k8s.gcr.io/etcd                                                      3.3.15-0            b2756210eeab        13 months ago       247MB

k8s.gcr.io/coredns                                                   1.6.2               bf261d157914        14 months ago       44.1MB

k8s.gcr.io/pause                                                     3.1                 da86e6ba6ca1        2 years ago         742kB

12 初始化master

此操作只在master01上执行即可

12.1 修改kubeadm配置文件

#apiserver里面写所有apiserver的主机名，ip地址或者后期可能加的ip

#controlPlaneEndpoint:vip地址

#podSubnet: "10.244.0.0/16"，后期网络的网段

[root@master01 ~]#more kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2

kind: ClusterConfiguration

kubernetesVersion: v1.16.4

apiServer:

 certSANs:    

 - master01

 - master02

 - master03

 - work01

 - work02

 - work03

 - 192.168.59.10

 - 192.168.59.11

 - 192.168.59.12

 - 192.168.59.13

 - 192.168.59.21

 - 192.168.59.22

controlPlaneEndpoint: "192.168.59.10:6443"

networking:

 podSubnet: "10.244.0.0/16"

12.2 初始化

#初始化，并且把内容导入到一个k8s_install.log的文件，方便后期加入节点的时候查看秘钥

[root@master01 ~]#kubeadm init --config=kubeadm-config.yaml | tee k8s_install.log

12.3 初始化后操作

[root@master01 ~]#mkdir -p $HOME/.kube

[root@master01 ~]#sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

[root@master01 ~]#sudo chown $(id -u):$(id -g) $HOME/.kube/config

12.4 加载环境变量

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

source .bash_profile

13 安装flannel

只在master01上执行

[root@master01 ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

[root@master01 ~]#kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.ymlpodsecuritypolicy.policy/psp.flannel.unprivileged created

clusterrole.rbac.authorization.k8s.io/flannel created

clusterrolebinding.rbac.authorization.k8s.io/flannel created

serviceaccount/flannel created

configmap/kube-flannel-cfg created

daemonset.apps/kube-flannel-ds-amd64 created

daemonset.apps/kube-flannel-ds-arm64 created

daemonset.apps/kube-flannel-ds-arm created

daemonset.apps/kube-flannel-ds-ppc64le created

daemonset.apps/kube-flannel-ds-s390x created

安装完flannel之后，其实单节点apiserver的集群已经好了，可以尝试检查下

[root@master01 ~]#kubectl get pods -n kube-system

NAME                               READY   STATUS    RESTARTS   AGE

coredns-5644d7b6d9-lkq9x           1/1     Running   0          4m59s

coredns-5644d7b6d9-rjtl8           1/1     Running   0          4m59s

etcd-master01                      1/1     Running   0          4m3s

kube-apiserver-master01            1/1     Running   0          3m59s

kube-controller-manager-master01   1/1     Running   0          3m49s

kube-flannel-ds-amd64-mcccp        1/1     Running   0          57s

kube-proxy-l48f6                   1/1     Running   0          4m59s

kube-scheduler-master01            1/1     Running   0          3m51s

14 添加其他master节点

master02 master03上

14.1 拷贝证书

需要将证书拷贝到其他的节点，才能加入集群

cd /etc/kubernetes/pki/

#拷贝6个证书

scp /etc/kubernetes/pki/ca.crt ca.key sa.key sa.pub front-proxy-ca.crt front-proxy-ca.key root@master02:/root/

scp /etc/kubernetes/pki/ca.crt ca.key sa.key sa.pub front-proxy-ca.crt front-proxy-ca.key root@master02:/root/

#拷贝etcd的两个证书

#先在目标主机创建两个目录

[root@master02 ~]#mkdir etcd

[root@master03 ~]#mkdir etcd

[root@master01 /etc/kubernetes/pki]#scp /etc/kubernetes/pki/etcd/ca.* root@master02:/root/etcd

[root@master01 /etc/kubernetes/pki]#scp /etc/kubernetes/pki/etcd/ca.* root@master03:/root/etcd

14.2 移动证书

需要在master02,03上把刚才拷贝的证书，移动到应该在的目录

#创建证书目录

[root@master02 ~]#mkdir -p /etc/kubernetes/pki/etcd

[root@master03 ~]#mkdir -p /etc/kubernetes/pki/etcd

#移动证书

[root@master03 ~]#mv ca.* sa.* front-* /etc/kubernetes/pki/

[root@master03 ~]#mv etcd/ca.* /etc/kubernetes/pki/etcd/

[root@master02 ~]#mv ca.* sa.* front-* /etc/kubernetes/pki/

[root@master02 ~]#mv etcd/ca.* /etc/kubernetes/pki/etcd/

14.3 master02加入集群

#查看master01初始化时候生成的日志文件

You can now join any number of control-plane nodes by copying certificate authorities

and service account keys on each node and then running the following as root:

 kubeadm join 192.168.59.10:6443 --token 1l0i97.wf409vm48u2qlb77 \

   --discovery-token-ca-cert-hash sha256:db8729d532f23e25c63df17f7cfaca073778f4ea5d4e0c88a9392b61b4a0eba0 \

   --control-plane      

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.59.10:6443 --token 1l0i97.wf409vm48u2qlb77 \

   --discovery-token-ca-cert-hash sha256:db8729d532f23e25c63df17f7cfaca073778f4ea5d4e0c88a9392b61b4a0eba0

开始初始化

#执行之前需要给docker镜像改名，还是之前的操作

[root@master02 ~]#docker images | grep k8s

k8s.gcr.io/kube-apiserver                                            v1.16.4             3722a80984a0        10 months ago       217MB

k8s.gcr.io/kube-controller-manager                                   v1.16.4             fb4cca6b4e4c        10 months ago       163MB

k8s.gcr.io/kube-proxy                                                v1.16.4             091df896d78f        10 months ago       86.1MB

k8s.gcr.io/kube-scheduler                                            v1.16.4             2984964036c8        10 months ago       87.3MB

k8s.gcr.io/etcd                                                      3.3.15-0            b2756210eeab        13 months ago       247MB

k8s.gcr.io/coredns                                                   1.6.2               bf261d157914        14 months ago       44.1MB

k8s.gcr.io/pause

kubeadm join 192.168.59.10:6443 --token 1l0i97.wf409vm48u2qlb77     --discovery-token-ca-cert-hash sha256:db8729d532f23e25c63df17f7cfaca073778f4ea5d4e0c88a9392b61b4a0eba0     --control-plane

14.4 master03加入集群

执行同上的操作

14.5 两台maseter初始化后

master02,master03上执行

To start administering your cluster from this node, you need to run the following as a regular user:

#执行下面操作

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

master01上操作

#把master01上的adminconf配置文件拷贝到master02,03上

[root@master01 ~]#scp /etc/kubernetes/admin.conf master02:/etc/kubernetes/

 

[root@master01 ~]#scp /etc/kubernetes/admin.conf master03:/etc/kubernetes/

master02,master03上执行

#这样做的目的是让master02,03可以执行kubectl命令

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

source .bash_profile

15 验证集群

#master01上

[root@master01 ~]#kubectl get nodes

NAME       STATUS   ROLES    AGE     VERSION

master01   Ready    master   41m     v1.16.4

master02   Ready    master   4m53s   v1.16.4

master03   Ready    master   4m53s   v1.16.4

#master02上

[root@master02 ~]#kubectl get nodes

NAME       STATUS   ROLES    AGE     VERSION

master01   Ready    master   41m     v1.16.4

master02   Ready    master   4m53s   v1.16.4

master03   Ready    master   4m53s   v1.16.4

#master03上

[root@master03 ~]#kubectl get nodes

NAME       STATUS   ROLES    AGE     VERSION

master01   Ready    master   41m     v1.16.4

master02   Ready    master   4m53s   v1.16.4

master03   Ready    master   4m53s   v1.16.4

[root@master01 ~]#kubectl get po -o wide -n kube-system

NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE       NOMINATED NODE   READINESS GATES

coredns-5644d7b6d9-lkq9x           1/1     Running   0          41m     10.244.0.2      master01   <none>           <none>

coredns-5644d7b6d9-rjtl8           1/1     Running   0          41m     10.244.0.3      master01   <none>           <none>

etcd-master01                      1/1     Running   0          40m     192.168.59.11   master01   <none>           <none>

etcd-master02                      1/1     Running   0          5m36s   192.168.59.12   master02   <none>           <none>

etcd-master03                      1/1     Running   0          5m21s   192.168.59.13   master03   <none>           <none>

kube-apiserver-master01            1/1     Running   0          40m     192.168.59.11   master01   <none>           <none>

kube-apiserver-master02            1/1     Running   0          5m36s   192.168.59.12   master02   <none>           <none>

kube-apiserver-master03            1/1     Running   0          4m19s   192.168.59.13   master03   <none>           <none>

kube-controller-manager-master01   1/1     Running   1          40m     192.168.59.11   master01   <none>           <none>

kube-controller-manager-master02   1/1     Running   0          5m36s   192.168.59.12   master02   <none>           <none>

kube-controller-manager-master03   1/1     Running   0          4m30s   192.168.59.13   master03   <none>           <none>

kube-flannel-ds-amd64-6v67w        1/1     Running   0          5m29s   192.168.59.13   master03   <none>           <none>

kube-flannel-ds-amd64-9c75g        1/1     Running   0          5m37s   192.168.59.12   master02   <none>           <none>

kube-flannel-ds-amd64-mcccp        1/1     Running   0          37m     192.168.59.11   master01   <none>           <none>

kube-proxy-4mxlf                   1/1     Running   0          5m29s   192.168.59.13   master03   <none>           <none>

kube-proxy-hrlsn                   1/1     Running   0          5m37s   192.168.59.12   master02   <none>           <none>

kube-proxy-l48f6                   1/1     Running   0          41m     192.168.59.11   master01   <none>           <none>

kube-scheduler-master01            1/1     Running   1          40m     192.168.59.11   master01   <none>           <none>

kube-scheduler-master02            1/1     Running   0          5m36s   192.168.59.12   master02   <none>           <none>

kube-scheduler-master03            1/1     Running   0          4m33s   192.168.59.13   master03   <none>           <none>

可以看到有3个etcd，3个apiserver，3个schedule，3个cm，3个flannel，3个proxy，2个dns

16 添加work节点

16.1 查看添加命令

kubeadm join 192.168.59.10:6443 --token 1l0i97.wf409vm48u2qlb77 \

   --discovery-token-ca-cert-hash sha256:db8729d532f23e25c63df17f7cfaca073778f4ea5d4e0c88a9392b61b4a0eba0

16.2 在work01上添加

kubeadm join 192.168.59.10:6443 --token 1l0i97.wf409vm48u2qlb77     --discovery-token-ca-cert-hash sha256:db8729d532f23e25c63df17f7cfaca073778f4ea5d4e0c88a9392b61b4a0eba0

16.3 在work02上添加

kubeadm join 192.168.59.10:6443 --token 1l0i97.wf409vm48u2qlb77     --discovery-token-ca-cert-hash sha256:db8729d532f23e25c63df17f7cfaca073778f4ea5d4e0c88a9392b61b4a0eba0

17 查看集群

发现flannel，proxy都是无法创建，查看日志，是无法下载镜像，想到了之前下载的镜像没有修改名字，于是，在work01和work02上修改名字，就可以了

18 安装dashboard

这里在work01上安装dashboard

18.1 下载yaml文件

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

18.2 修改yaml文件

#因为默认的yum源的地址不通，换成阿里云

sed -i 's/kubernetesui/registry.cn-hangzhou.aliyuncs.com\/loong576/g' recommended.yaml

#修改端口

sed -i '/targetPort: 8443/a\ \ \ \ \ \ nodePort: 30001\n\ \ type: NodePort' recommended.yaml

cat >> recommended.yaml << EOF

---

# ------------------- dashboard-admin ------------------- #

apiVersion: v1

kind: ServiceAccount

metadata:

 name: dashboard-admin

 namespace: kubernetes-dashboard

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding

metadata:

 name: dashboard-admin

subjects:

- kind: ServiceAccount

 name: dashboard-admin

 namespace: kubernetes-dashboard

roleRef:

 apiGroup: rbac.authorization.k8s.io

 kind: ClusterRole

 name: cluster-admin

EOF

18.3 启动dashborad

[root@work01 ~]#kubectl apply -f recommended.yaml

namespace/kubernetes-dashboard created

serviceaccount/kubernetes-dashboard created

service/kubernetes-dashboard created

secret/kubernetes-dashboard-certs created

secret/kubernetes-dashboard-csrf created

secret/kubernetes-dashboard-key-holder created

configmap/kubernetes-dashboard-settings created

role.rbac.authorization.k8s.io/kubernetes-dashboard created

clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created

rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created

clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created

deployment.apps/kubernetes-dashboard created

service/dashboard-metrics-scraper created

deployment.apps/dashboard-metrics-scraper created

18.3 查看状态

[root@work01 ~]#kubectl get all -n kubernetes-dashboard

18.4 查看token

kubectl describe secrets -n kubernetes-dashboard dashboard-admin

token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IldBNTktTWUxVUtTMm1BaWNzeEE2eFZWcGEtMjlZMDlfLUt4WmJpWEMtYlUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tZDU4NmQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNzY0NzRjNWItNWUzYy00MDFhLWI2NzktYWVlMmRlYjg2MzQ3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmVybmV0ZXMtZGFzaGJvYXJkOmRhc2hib2FyZC1hZG1pbiJ9.CkJVRmU_41JAqHa-jzl8m-Jzh7qr4Ct-jXY-LUzAg0ilR48wVQHUl48D1j-eHYU5_POdSgsoEJwGD77gy8AeDgjiF6BHbknRci5z-XA3x1WmMkziTkjUjRC2hsvi81zGSDRCgfP4iNotggg361yXbhokjwq82W6jWPSUskOvttpVAN7px3hc34bjvMJTWXaoAtWem29BGoi-FjQUF2nOJD5JqoKO7k5LNwgylMWqcsMeNDU9aJQSWy3axZP7BEUnhPiCMi94MjNeqYXDrREZO0GWvJVvGF8V8lj_w0TdTDwp8zJoZqe6N2IdWeRNc834949EACAHKrizXQO0QdccZg

19 故障模拟

19.1 关闭master01设备

#先查看调度器在哪个节点

[root@master01 ~]kubectl get endpoints kube-scheduler -n kube-system

#关闭master01

[root@master01 ~]#init 0

#在查看

[root@master02 ~]#kubectl get endpoints kube-scheduler -n kube-system -o yaml |grep holderIdentity

   control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"master02_a802b7d7-27df-46fb-8422-3c08115fb76f","leaseDurationSeconds":15,"acquireTime":"2020-10-27T03:59:37Z","renewTime":"2020-10-27T04:01:19Z","leaderTransitions":2}'

#查看集群节点

[root@master02 ~]#kubectl get nodes

NAME       STATUS     ROLES    AGE   VERSION

master01   NotReady   master   88m   v1.16.4

master02   Ready      master   51m   v1.16.4

master03   Ready      master   51m   v1.16.4

work01     Ready      <none>   42m   v1.16.4

work02     Ready      <none>   31m   v1.16.4

#master01已经是not ready了

#此时vip已经飘到master02上了

20 创建pod

#编辑nginx.yaml文件

apiVersion: apps/v1             #描述文件遵循extensions/v1beta1版本的Kubernetes API

kind: Deployment                #创建资源类型为Deployment

metadata:                       #该资源元数据

 name: nginx-master            #Deployment名称

spec:                           #Deployment的规格说明

 selector:

   matchLabels:

     app: nginx

 replicas: 3                   #指定副本数为3

 template:                     #定义Pod的模板

   metadata:                   #定义Pod的元数据

     labels:                   #定义label（标签）

       app: nginx              #label的key和value分别为app和nginx

   spec:                       #Pod的规格说明

     containers:              

     - name: nginx             #容器的名称

       image: nginx:latest     #创建容器所使用的镜像

kubectl apply -f nginx.yaml

kubectl get po -o wide