基本概念的理解
k8s是一种编排工具，类似于docker-compose，但是应用比后者广泛。
k8s水平扩展访问，本质上是增加pod，且新增的pod均匀分布在不同的机器上。
概念的层级关系k8s–node(对应一台物理机器)–pod。
容器有docker，rocket。k8s与docker的组合最常用。
pod是k8s最小调度单元。每个pod有自己独立的ip。一个pod里面多个docker container的ip都是一样的。pod里面可以开多个docker container。一般情况下一个pod只开一个container，当同一个pod开多个container的时候，多个容器的协作关系采用localhost的方式通信。
节点中的kubelet相当于node server的作用，会上报节点中不同pod的信息。
kublet-pod-docker-container三者之间的关系，类似操作系统中的调度器-进程-线程之间的关系。
所有的数据交互都是通过server-api组件来完成。
etcd默认在master节点上。
service是用来统一管理pod的组件，怎样将pod分发到不同的node。对外表现为一个访问入口，可以认为是一个用来做负载均衡的网管。
参考书：《k8s权威指南》
实操
准备工作
1. 禁用swap
sudo swapoff -a #无输出
注释掉/etc/fstab中关于swap语句
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-UmKR3G1L-1664502154890)(D:\Documents\fiberhome\MarkDown\Images\0voice\2022-09-19 194506.png)]
2. 关闭防火墙
master@ubuntu:~$ sudo systemctl stop firewalld #提示防火墙服务没有导入 Failed to stop firewalld.service: Unit firewalld.service not loaded. master@ubuntu:~$ sudo systemctl disable firewalld Failed to disable unit: Unit file firewalld.service does not exist.
3. 禁用selinux
更新ubuntu源
sudo apt install selinux-utils #安装不成功，跟mysql有关系 E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.
这里属于历史遗留问题，参考另一篇解决方法。解决后，更新软件源，用来更快安装selinux。
sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak
更新文件内容，用阿里源替换
sudo gedit /etc/apt/sources.list # 阿里源 deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
selinux安装完成后，执行
master@ubuntu:~$ sudo setenforce 0 setenforce: SELinux is disabled
4. 修改主机名
master@ubuntu:~$ sudo vim /etc/hosts master@ubuntu:~$ sudo /etc/init.d/networking restart [ ok ] Restarting networking (via systemctl): networking.service.
设置主机名
sudo hostnamectl set-hostname qiu-k8s-m
键入hostnamectl显示主机名已修改成功
master@ubuntu:~$ hostnamectl Static hostname: qiu-k8s-m Icon name: computer-vm Chassis: vm ... ...
经过重启后，终端中会显示改过后的主机名
master@qiu-k8s-m:~$
5. 安装docker
已经安装，输入docker -v可显示版本
master@qiu-k8s-m:~$ docker -v Docker version 20.10.17, build 100c701
将当前用户加入docker组，避免每次docker命令都用sudo。但是要使修改后的/etc/group生效得重启机器。
sudo groupadd docker #打开/etc/group最后一行会有docker:x:GID: sudo usermod -aG docker $USER #将当前用户加入到GID:后面
测试docker能否正常拉取镜像
master@qiu-k8s-m:~$ docker run -it ubuntu bash Unable to find image 'ubuntu:latest' locally latest: Pulling from library/ubuntu 2b55860d4c66: Pull complete Digest: sha256:20fa2d7bb4de7723f542be5923b06c4d704370f0390e4ae9e1c833c8785644c1 Status: Downloaded newer image for ubuntu:latest root@32cfec738c47:/#
能正常显示以上信息，则表示docker安装成功。
如果拉取速度太慢，是因为要从Docker Hub上下载，可修改相关文件
sudo vim /etc/docker/daemon.json #添加 { "registry-mirrors": ["https://alzgoonw.mirror.aliyuncs.com"], "live-restore": true }
并且执行
sudo systemctl daemon-reload sudo systemctl restart docker
来重启docker服务。更新后再拉取镜像就快很多了。
如果启动docker服务出现问题，可能是/etc/docker/daemon.json改的有问题，使用dockerd --debug排查
master@qiu-k8s-m:~$ sudo dockerd --debug unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives don't match any configuration option: __registry-mirrors
准备安装kubectl，kubelet，kubeadm
添加秘钥
master@qiu-k8s-m:~$ curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add - % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2537 100 2537 0 0 898 0 0:00:02 0:00:02 --:--:-- 898 OK
添加Kubernetes软件源，采用ustc源
sudo vim /etc/apt/source.list.d/kubernetes.list
添加内容
deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
由于xenial是ubuntu16.04的代号，而实验的机器是18.04，理应将xenial替换成bionic，但换过之后反而找不到源，于是再换回来。
安装kubelet，kubelet，kubeadm
更新源，然后使用apt-get install的方式安装
sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo systemctl enable kubelet
查看节点试试
master@qiu-k8s-m:~$ kubectl get nodes The connection to the server localhost:8080 was refused - did you specify the right host or port?
或者报这个错误
master@qiu-k8s-m:~$ sudo kubectl get nodes Unable to connect to the server: dial tcp 192.168.230.246:6443: connect: no route to host
因为还没有布置节点，所以无节点显示。
将其他两个节点按同样的步骤过一遍。
配置master
配置环境变量
sudo vim /etc/profile
末尾加入
export KUBECONFIG=/etc/kubernetes/admin.conf
立即生效
source /etc/profile
重启kubelet
sudo systemctl daemon-reload sudo systemctl restart kubelet
初始化kubeadm
sudo kubeadm init --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.230.246 --ignore-preflight-errors=NumCPU
报错
W0921 11:13:33.282440 3157 version.go:104] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers) W0921 11:13:33.282636 3157 version.go:105] falling back to the local client version: v1.25.1 [init] Using Kubernetes version: v1.25.1 [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR CRI]: container runtime is not running: output: E0921 11:13:33.473505 3198 remote_runtime.go:948] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" time="2022-09-21T11:13:33+08:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" , error: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher
解决办法，删掉/etc/containerd/config.toml文件，重启容器服务
sudo rm -rf /etc/containerd/config.toml systemctl restart containerd
可以跑起来，但是很慢，不过提示可以先执行sudo kubeadm config images pull这个命令
master@qiu-k8s-m:~$ sudo kubeadm init --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.230.246 --ignore-preflight-errors=NumCPU [sudo] password for master: [init] Using Kubernetes version: v1.25.1 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' ^C
于是先拉镜像，同样很慢，不过还是熬结束了
master@qiu-k8s-m:~$ sudo kubeadm config images pull --image-repository registry.aliyuncs.com/google_containers [config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.25.1 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.25.1 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.25.1 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.25.1 [config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.8 [config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.4-0 [config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.9.3
再回来执行init那条语句，但是遇到错误
... [kubelet-check] Initial timeout of 40s passed. Unfortunately, an error has occurred: timed out waiting for the condition This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet' ...
说是kubelet没有跑起来？那就再执行下reload
error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists [ERROR Port-10250]: Port 10250 is in use
仍然报错，显示端口已在使用。键入kubeadm reset，会删掉一些东西。重新执行init语句，会在
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed
等好一会，然后重新出现
Unfortunately, an error has occurred: timed out waiting for the condition ...
查来查去，可能是之前删除/etc/containerd/config.toml文件导致，因此再新建，加入一些信息
先在当前目录生成这个文件
containerd config default > config.toml
打开文件，将k8s.gcr.io替换为registry.cn-hangzhou.aliyuncs.com/google_containers
60 restrict_oom_score_adj = false 61 sandbox_image = "k8s.gcr.io/pause:3.6" 62 selinux_category_range = 1024
改完后
60 restrict_oom_score_adj = false 61 sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6" 62 selinux_category_range = 1024
将改完后的config.toml文件移动到/etc/containerd/下（当然也可以直接在/etc/containerd/目录下建文件，再修改）
再次执行init那句，仍然报错
.... This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet' ....
于是决定按照提示在init这句后加上–v=6来获取更详细的信息，可以看到
... I0926 22:36:31.414225 86362 round_trippers.go:553] GET https://192.168.230.246:6443/healthz?timeout=10s in 2720 milliseconds I0926 22:36:34.485995 86362 round_trippers.go:553] GET https://192.168.230.246:6443/healthz?timeout=10s in 2792 milliseconds I0926 22:36:37.557797 86362 round_trippers.go:553] GET https://192.168.230.246:6443/healthz?timeout=10s in 2863 milliseconds [kubelet-check] Initial timeout of 40s passed. I0926 22:36:40.630317 86362 round_trippers.go:553] GET https://192.168.230.246:6443/healthz?timeout=10s in 2935 milliseconds I0926 22:36:43.701762 86362 round_trippers.go:553] GET https://192.168.230.246:6443/healthz?timeout=10s in 3007 milliseconds ...
一直重复出现这几句，显然是健康检查时，连接主机地址超时了。通过ifconfig查看自己的ip已经改变，输入了一个不可用的ip，更改后，sudo kubeadm reset，再次init，终于出现了想要的输出
... [addons] Applied essential addon: kube-proxy I0926 22:45:05.364595 88564 loader.go:374] Config loaded from file: /etc/kubernetes/admin.conf I0926 22:45:05.366180 88564 loader.go:374] Config loaded from file: /etc/kubernetes/admin.conf Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.230.130:6443 --token 65rht0.rs2weic09flv5o8s \ --discovery-token-ca-cert-hash sha256:193cc3031acdde64264aba3b186e25462a2c8707b6d9bc27ba92393962d5eece
如果忘记保存join节点的token值，可输入命令
sudo kubeadm token create --print-join-command
根据以上提示需要输入
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
接着安装flannel网络组件
sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
稍等一会执行可以看到ready的状态
master@qiu-k8s-m:~$ sudo kubectl get node NAME STATUS ROLES AGE VERSION qiu-k8s-m Ready control-plane 20m v1.25.1
执行init的过程中，由于旧的命令导致端口在使用，可以reset重设
[init] Using Kubernetes version: v1.25.2 [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR Port-6443]: Port 6443 is in use [ERROR Port-10259]: Port 10259 is in use [ERROR Port-10257]: Port 10257 is in use [ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists [ERROR Port-10250]: Port 10250 is in use [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher
端口被占用，重置试试
sudo kubeadm reset
重启kubeadm可以解决这个问题。
从节点配置
执行join那句报错
[preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR CRI]: container runtime is not running: output: E0927 21:58:31.253378 27048 remote_runtime.go:948] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" time="2022-09-27T21:58:31+08:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" , error: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher
初步判断是container有关的错误，查到这个资料，解决办法为
rm -rf /etc/containerd/config.toml systemctl restart containerd
后再执行join那句，最后能看到这个提示
... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ...
表示把当前node加入到集群了，但是输入get nodes这句，仍然有报错
The connection to the server localhost:8080 was refused - did you specify the right host or port?
参考这篇，将主节点/etc/kubenetes/admin.conf拷贝到从节点的指定目录
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
再次执行，可看到节点信息列表
NAME STATUS ROLES AGE VERSION qiu-k8s-m Ready control-plane 82m v1.25.1 qiu-k8s-n2 NotReady <none> 10m v1.25.1
最后安装flannel网络组件
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
可能是网络连接错误
The connection to the server raw.githubusercontent.com was refused - did you specify the right host or port?
那么换种方式，将主节点的/etc/cni拷贝到从节点下的相同目录
cp -r /etc/cni /mnt/ sudo mv /mnt/cni /etc/
告一段落。
yaml文件里kind有四种类型，Pod，ReolicationController，Deployment，Service。
yaml文件tab和空格不能混用，不支持tab缩进。NodePort是对外提供的端口。
重新配置master节点和从节点，有错误如下
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
这是由于在sudo kubeadm reset时没有删除~/.kube/config文件所致(因为这个文件是手动拷过去的)，解决办法重新拷一份
sudo cp /etc/kubernetes/admin.conf ~/.kube/config
从节点~/.kube/config也需要从主节点拷过去。
为什么配的从节点没有内部的ip?
通过查看pod详情
sudo kubectl describe pod httpd
错误细节
坑太多。。。未完待续
