🛠️ 实战环节！基于 K3S + WireGuard + Kilo 搭建跨多云的统一 K8S 集群。💪💪💪
步骤
1. 前提
1.1 跨云的多台云主机
准备至少 2 台不同公有云的云主机（配置最低 1C1G 就能跑得动），这里准备了 6 台，主机名要求各不相同，分别是：
- 天翼云：
ty1（K3S Server） - 阿里云：
ali（K3S Agent） - 华为云：
hw1（K3S Agent） - 百度云：
bd1和bd2（K3S Agent） - 腾讯云：
tx1（K3S Agent）
1.2 操作系统
操作系统：推荐 ubuntu 20.04（截止 2022/1/22），因为可以安装 WireGuard 方便。
1.3 已安装 WireGuard
且已安装 WireGuard，安装过程见👉️这里
1.4 网络
| 协议 | 端口 | 源 | 描述 |
| TCP | 6443 | K3s agent 节点 | Kubernetes API Server |
| UDP | 51820 | K3s server 和 agent 节点 | Kilo 网络间通信 |
| TCP | 10250 | K3s server 和 agent 节点 | Kubelet metrics |
| TCP | 2379-2380 | K3s server 节点 | 只有嵌入式 etcd 高可用才需要 |
通常情况下，所有出站流量都是允许的。
1.5 确保云主机有公网 IP
每个 location 至少有一个节点的 IP 地址与其他位置路由可达。
如果 location 在不同的云或私有网络中，那么这必须是一个公共 IP 地址。如果该 IP 地址没有在节点的以太网设备上自动配置，可以使用 kilo.squat.ai/force-endpoint 注释方式手动指定。
1.6 云主机启用 IPv4 Forwarding
在 /etc/sysctl.conf 有这么一行：
net.ipv4.ip_forward=1 INI |
并生效：systel -p
1.7 （可选）配置镜像仓库配置
参考这里👉️：私有镜像仓库配置参考
主要是为了加速 pull 镜像。
2. 安装 K3S Server
备注：
K3S Server 可以简单的类比为 K8S 的 master。
2.1 一键安装 K3S Server
细节省略，直接通过 cnrancher 的安装脚本来一键安装：
curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_CLUSTER_INIT=true INSTALL_K3S_EXEC="--tls-san {{ server_public_ip }} --node-external-ip {{ master_ip }} --flannel-backend none --kube-proxy-arg metrics-bind-address=0.0.0.0 --kube-apiserver-arg feature-gates=EphemeralContainers=true" sh -s - BASH |
简要说明：
K3S_CLUSTER_INIT=true: 集群模式，会安装内置的 etcd，而不是 sqlite3；--tls-san {{ server_public_ip }}:{{ server_public_ip }}改为你自己指定的 k3s server 公网 IP，--tls-san这个选项在 TLS 证书中增加了一个额外的主机名或 IP 作为备用名称，如果你想通过 IP 和主机名访问，可以多次指定。--node-external-ip {{ server_public_ip }}：指定 node 的公网 IP--flannel-backend none：K3S 默认网络插件为 flannel，这里意思是不使用 flannel，后面单独安装 Kilo。--kube-proxy-arg metrics-bind-address=0.0.0.0: kube-proxy 参数。- （可选）：
--kube-apiserver-arg feature-gates=EphemeralContainers=true启用 feature-gates：EphemeralContainers，方便在不重启 pod 的情况下附加 sidecar 进行调试。
❌ 错误 ：
- 如果不指定
--tls-san，可能会导致kubectl无法通过server_public_ip来访问集群。 - 如果不指定
--node-external-ip，可能会导致位于其他云的 k3s agent 无法连接到 K3S Server 的 API。 - 如果不指定
--kube-proxy-arg metrics-bind-address=0.0.0.0，可能导致无法获取到 Metrics。
2.2 查看安装结果
具体如下：
❯ systemctl status k3s.service ● k3s.service - Lightweight Kubernetes Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2022-01-22 16:27:14 CST; 4h 5min ago Docs: https://k3s.io Process: 5757 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS) Process: 5758 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Main PID: 5759 (k3s-server) Tasks: 49 Memory: 926.0M CGroup: /system.slice/k3s.service ├─ 5759 /usr/local/bin/k3s server ├─ 5774 containerd ├─18561 /var/lib/rancher/k3s/data/2e877cf4762c3c7df37cc556de3e08890fbf450914bb3ec042ad4f36b5a2413a/bin/containerd-shim-runc-v2 -namespace k> ├─18579 /pause └─18745 /opt/bin/kg --kubeconfig=/etc/kubernetes/kubeconfig --hostname=ty1.k3s Jan 22 17:28:41 ty1.k3s k3s[5759]: I0122 17:28:41.435542 5759 reconciler.go:319] "Volume detached for volume... ... SHELL |
❯ journalctl -f -b -u k3s.service -- Logs begin at Fri 2021-03-26 09:47:06 CST, end at Sat 2022-01-22 20:35:12 CST. -- Jan 22 16:20:21 ty1.k3s systemd[1]: Starting Lightweight Kubernetes... Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Starting k3s v1.22.5+k3s1 (405bf79d)" Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, max> Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Configuring database table schema and indexes, this may take a momen> Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Database tables and indexes are up to date" Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Kine available at unix://kine.sock" Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Reconciling bootstrap data between datastore and disk" Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Running kube-apiserver --advertise-address=x.x.x.x --adverti> Jan 22 16:20:22 ty1.k3s k3s[1660]: Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24. Jan 22 16:20:22 ty1.k3s k3s[1660]: I0122 16:20:22.307328 1660 server.go:581] external host was not specified, using x.x.x.x Jan 22 16:20:22 ty1.k3s k3s[1660]: I0122 16:20:22.308688 1660 server.go:175] Version: v1.22.5+k3s1 SHELL |
❯ k3s kubectl cluster-info Kubernetes control plane is running at https://127.0.0.1:6443 CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy SHELL |
❯ k3s kubectl get node NAME STATUS ROLES AGE VERSION ty1.k3s NotReady control-plane,master 4h10m v1.22.5+k3s1 BASH |
⚠️ 注意 ：
此时 K3S Server 的状态是 NotReady，这是正常状况，因为还没有安装 CNI 网络插件。
3. 安装 Kilo
3.1 指定 K3S Server 拓扑
我的 K3S Server 在天翼云上，指定拓扑：
k3s kubectl annotate node ty1.k3s kilo.squat.ai/location="ctyun" k3s kubectl annotate node ty1.k3s kilo.squat.ai/force-endpoint=x.x.x.x:51820 k3s kubectl annotate node ty1.k3s kilo.squat.ai/persistent-keepalive=20 BASH |
解释在这里👉️：Kilo Annotions - location
3.2 安装 Kilo❗️
通过在集群中部署一个 DaemonSet 来安装 Kilo。
kubectl apply -f https://gitee.com/mirrors/squat/raw/main/manifests/crds.yaml kubectl apply -f https://gitee.com/mirrors/squat/raw/main/manifests/kilo-k3s.yaml BASH |
ℹ️ 备注 ：
上面的地址是 Kilo 的 Gitee mirror 仓库。
详细说明：
crds.yaml安装了peer.kilo.squat.ai，用于通过该 CRD 配置 WireGuard Peer。kilo-k3s.yaml安装了 Kilo CNI，包括：
- ConfigMap
cni-conf.jsonkilo-scripts
- ServiceAccount
- ClusterRole
- ClusterRoleBinding
- DaemonSet: 运行在所有 Node 上
- 其中，WireGuard 的配置：
- ini 格式的 WireGuard 配置位于：
/var/lib/kilo/conf - WireGuard 的 key 位于：
/var/lib/kilo/key
3.3 验证
kube-systemns 中，已创建kiloDaemonSet，且所有 Pod 已出于 Running 状态- K3S Server Node 状态从
NotReady变为Ready - K3S Server Node 上，已经有 kilo 的相关 annotations, 示例如下：
apiVersion: v1 kind: Node metadata: name: ty1.k3s labels: beta.kubernetes.io/arch: amd64 ... annotations: k3s.io/external-ip: x.x.x.x k3s.io/hostname: ty1.k3s k3s.io/internal-ip: 192.168.1.226 k3s.io/node-args: >- ["server","--tls-san","x.x.x.x","--node-external-ip","x.x.x.x","--flannel-backend","none","--kube-proxy-arg","metrics-bind-address=0.0.0.0","--kube-apiserver-arg","feature-gates=EphemeralContainers=true"] kilo.squat.ai/endpoint: x.x.x.x:51820 kilo.squat.ai/force-endpoint: x.x.x.x:51820 kilo.squat.ai/granularity: location kilo.squat.ai/internal-ip: 192.168.1.226/24 kilo.squat.ai/key: zCiXXXXXXXXXXXXXXXXXXXXXXXXXXXXQTL9CEc= kilo.squat.ai/last-seen: '1642856638' kilo.squat.ai/location: ctyun kilo.squat.ai/persistent-keepalive: '20' kilo.squat.ai/wireguard-ip: 10.4.0.3/16 ... YAML |
4. 安装 K3S Agent
4.1 一键安装 K3S Agent
1 |
curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_TOKEN={{ token }} K3S_URL=https://{{ server_public_ip }}:6443 sh -s - --node-external-ip {{ node_public_ip }} --kube-proxy-arg "metrics-bind-address=0.0.0.0" BASH |
4.2 等待 K3S Agent 加入集群中
等待 K3S Agent 加入集群中，状态 NotReady 没关系：
❯ systemctl status k3s-agent.service ● k3s-agent.service - Lightweight Kubernetes Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2022-01-22 16:27:35 CST; 4h 44min ago Docs: https://k3s.io Process: 4079 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS) Process: 4080 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Main PID: 4081 (k3s-agent) Tasks: 63 Memory: 126.9M CGroup: /system.slice/k3s-agent.service ├─4081 /usr/local/bin/k3s agent ├─4106 containerd ├─5285 /var/lib/rancher/k3s/data/ ... -- Logs begin at Sat 2021-11-06 14:00:29 CST, end at Sat 2022-01-22 21:10:33 CST. -- Jan 22 16:27:35 ali1.k3s systemd[1]: Starting Lightweight Kubernetes... Jan 22 16:27:35 ali1.k3s systemd[1]: Started Lightweight Kubernetes. Jan 22 16:27:35 ali1.k3s k3s[4081]: time="2022-01-22T16:27:35+08:00" level=info msg="Starting k3s agent v1.22.5+k3s1 (405bf79d)" Jan 22 16:27:35 ali1.k3s k3s[4081]: time="2022-01-22T16:27:35+08:00" level=info msg="Running load balancer 127.0.0.1:6444 -> [192.168.1.226:6443 140.246.255.203:6443]" Jan 22 16:27:55 ali1.k3s k3s[4081]: time="2022-01-22T16:27:55+08:00" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jan 22 16:28:01 ali1.k3s k3s[4081]: time="2022-01-22T16:28:01+08:00" level=info msg="Module overlay was already loaded" Jan 22 16:28:01 ali1.k3s k3s[4081]: time="2022-01-22T16:28:01+08:00" level=info msg="Module nf_conntrack was already loaded" Jan 22 16:28:01 ali1.k3s k3s[4081]: time="2022-01-22T16:28:01+08:00" level=info msg="Module br_netfilter was already loaded" Jan 22 16:28:01 ali1.k3s k3s[4081]: time="2022-01-22T16:28:01+08:00" level=info msg="Module iptable_nat was already loaded" Jan 22 16:28:01 ali1.k3s k3s[4081]: time="2022-01-22T16:28:01+08:00" level=info msg="Using private registry config file at /etc/rancher/k3s/registries.yaml" Jan 22 16:28:01 ali1.k3s k3s[4081]: time="2022-01-22T16:28:01+08:00" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log" Jan 22 16:28:01 ali1.k3s k3s[4081]: time="2022-01-22T16:28:01+08:00" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd" Jan 22 16:28:02 ali1.k3s k3s[4081]: time="2022-01-22T16:28:02+08:00" level=info msg="Containerd is now running" Jan 22 16:28:02 ali1.k3s k3s[4081]: time="2022-01-22T16:28:02+08:00" level=info msg="Updating load balancer server addresses -> [140.246.255.203:6443]" Jan 22 16:28:02 ali1.k3s k3s[4081]: time="2022-01-22T16:28:02+08:00" level=info msg="Connecting to proxy" url="wss://140.246.255.203:6443/v1-k3s/connect" Jan 22 16:28:02 ali1.k3s k3s[4081]: time="2022-01-22T16:28:02+08:00" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/cli> Jan 22 16:28:02 ali1.k3s k3s[4081]: Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet. Jan 22 16:28:02 ali1.k3s k3s[4081]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed. Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.742554 4081 server.go:436] "Kubelet version" kubeletVersion="v1.22.5+k3s1" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.798379 4081 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.891854 4081 server.go:687] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.892197 4081 container_manager_linux.go:280] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.892310 4081 container_manager_linux.go:285] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot> Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.892345 4081 topology_manager.go:133] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.892357 4081 container_manager_linux.go:320] "Creating device plugin manager" devicePluginEnabled=true Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.892403 4081 state_mem.go:36] "Initialized new in-memory state store" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.893507 4081 kubelet.go:418] "Attempting to sync node with API server" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.893533 4081 kubelet.go:279] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.893577 4081 kubelet.go:290] "Adding apiserver pod source" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.893601 4081 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.897574 4081 kuberuntime_manager.go:245] "Container runtime initialized" containerRuntime="containerd" version="v1.5.8-k3s1" apiVersion="v1alpha2" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.898404 4081 server.go:1213] "Started kubelet" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.901202 4081 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.903670 4081 server.go:149] "Starting to listen" address="0.0.0.0" port=10250 Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.904727 4081 server.go:409] "Adding debug handlers to kubelet server" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.909003 4081 volume_manager.go:291] "Starting Kubelet Volume Manager" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.909628 4081 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.949667 4081 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv4 Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.971847 4081 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv6 Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.972256 4081 status_manager.go:158] "Starting to sync pod status with apiserver" Jan 22 16:28:02 ali1.k3s k3s[4081]: I0122 16:28:02.972423 4081 kubelet.go:1967] "Starting kubelet main sync loop" ... Jan 22 16:28:03 ali1.k3s k3s[4081]: I0122 16:28:03.331465 4081 kubelet_node_status.go:74] "Successfully registered node" node="ali1.k3s" BASH |
❯ k3s kubectl get node NAME STATUS ROLES AGE VERSION ali1.k3s NotReady worker 4h41m v1.22.5+k3s1 bd2.k3s NotReady worker 4h40m v1.22.5+k3s1 ty1.k3s Ready control-plane,master 4h42m v1.22.5+k3s1 tx1.k3s NotReady worker 4h41m v1.22.5+k3s1 hw1.k3s NotReady worker 4h41m v1.22.5+k3s1 bd1.k3s NotReady worker 4h40m v1.22.5+k3s1 BASH |
4.3 指定 K3S Agent 拓扑
# ali1 k3s kubectl annotate node ali1.k3s kilo.squat.ai/location="aliyun" k3s kubectl annotate node ali1.k3s kilo.squat.ai/force-endpoint={{ ali1_public_ip }}:51820 k3s kubectl annotate node ali1.k3s kilo.squat.ai/persistent-keepalive=20 # hw1 k3s kubectl annotate node hw1.k3s kilo.squat.ai/location="huaweicloud" k3s kubectl annotate node hw1.k3s kilo.squat.ai/force-endpoint={{ hw1_public_ip }}:51820 k3s kubectl annotate node hw1.k3s kilo.squat.ai/persistent-keepalive=20 # bd1 k3s kubectl annotate node bd1.k3s kilo.squat.ai/location="baidu" k3s kubectl annotate node bd1.k3s kilo.squat.ai/force-endpoint={{ bd1_public_ip }}:51820 k3s kubectl annotate node bd1.k3s kilo.squat.ai/persistent-keepalive=20 # bd2 k3s kubectl annotate node bd2.k3s kilo.squat.ai/location="baidu" k3s kubectl annotate node bd2.k3s kilo.squat.ai/force-endpoint={{ bd2_public_ip }}:51820 k3s kubectl annotate node bd2.k3s kilo.squat.ai/persistent-keepalive=20 # tx1 k3s kubectl annotate node tx1.k3s kilo.squat.ai/location="tencentcloud" k3s kubectl annotate node tx1.k3s kilo.squat.ai/force-endpoint={{ tx2_public_ip }}:51820 k3s kubectl annotate node tx1.k3s kilo.squat.ai/persistent-keepalive=20 BASH |
等待所有的节点状态 Ready.
5. 验证
5.1 验证网络通断
部署 1 个 busybox 的 DaemonSet，使其落在每一个 Node 上。验证网络流向：
进入其中一个 pod，ping 其他 pod，可以互相 ping 通，示例如下：
/ # ping 10.42.2.7 PING 10.42.2.7 (10.42.2.7): 56 data bytes 64 bytes from 10.42.2.7: seq=0 ttl=62 time=6.604 ms 64 bytes from 10.42.2.7: seq=1 ttl=62 time=6.520 ms 64 bytes from 10.42.2.7: seq=2 ttl=62 time=6.412 ms 64 bytes from 10.42.2.7: seq=3 ttl=62 time=6.430 ms 64 bytes from 10.42.2.7: seq=4 ttl=62 time=6.487 ms ^C --- 10.42.2.7 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 6.412/6.490/6.604 ms / # ping 10.42.1.3 PING 10.42.1.3 (10.42.1.3): 56 data bytes 64 bytes from 10.42.1.3: seq=0 ttl=62 time=7.426 ms 64 bytes from 10.42.1.3: seq=1 ttl=62 time=7.123 ms 64 bytes from 10.42.1.3: seq=2 ttl=62 time=7.109 ms 64 bytes from 10.42.1.3: seq=3 ttl=62 time=7.129 ms ^C --- 10.42.1.3 ping statistics --- 4 packets transmitted, 4 packets received, 0% packet loss round-trip min/avg/max = 7.109/7.196/7.426 m BASH |
ping 其他 node 的 WireGuard 地址，也可以 ping 通，如下：
/ # ping 10.4.0.1 PING 10.4.0.1 (10.4.0.1): 56 data bytes 64 bytes from 10.4.0.1: seq=0 ttl=64 time=0.077 ms 64 bytes from 10.4.0.1: seq=1 ttl=64 time=0.099 ms ^C --- 10.4.0.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.077/0.088/0.099 ms / # ping 10.4.0.2 PING 10.4.0.2 (10.4.0.2): 56 data bytes 64 bytes from 10.4.0.2: seq=0 ttl=63 time=29.000 ms 64 bytes from 10.4.0.2: seq=1 ttl=63 time=28.939 ms ^C --- 10.4.0.2 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 28.939/28.969/29.000 ms BASH |
ping 其他 node 的默认内网地址，也可以 ping 通，如下：
/ # ping 172.17.0.3 PING 172.17.0.3 (172.17.0.3): 56 data bytes 64 bytes from 172.17.0.3: seq=0 ttl=63 time=6.327 ms 64 bytes from 172.17.0.3: seq=1 ttl=63 time=6.350 ms ^C --- 172.17.0.3 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 6.327/6.338/6.350 ms / # ping 192.168.64.4 PING 192.168.64.4 (192.168.64.4): 56 data bytes 64 bytes from 192.168.64.4: seq=0 ttl=63 time=29.261 ms 64 bytes from 192.168.64.4: seq=1 ttl=63 time=29.015 ms ^C --- 192.168.64.4 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 29.015/29.138/29.261 ms BASH |
证明网络（pod-pod，pod-node）全通。
5.2 ℹ️ 详细说明
5.2.1 网卡
Kilo 建立了 3 个网卡，分别为：
kilo0：WireGuard VPN 网络，用以 Node 间组建 VPN 内网；（bd2上没有该网卡，因为和bd1同属一个 VPC 内网，内网内连接默认不通过 WireGuard 加密，只有出 VPC 内网才会走 WireGuard 加密）kube-bridge: 桥接网络，使得 Pod 的网卡与云主机的网卡进行连接，以此实现 Pod 通过 WireGuard VPN 网络共享通信；
tunl0：Bridge 模式下，多主机网络通信需要额外配置主机路由，或使用 overlay 网络。通过 Kilo 来自动配置。比如 overlay 情况下的网络结构为：
其中，kube-bridge 配置如下：
{ "cniVersion": "0.3.1", "name": "kilo", "plugins": [ { "name": "kubernetes", "type": "bridge", "bridge": "kube-bridge", "isDefaultGateway": true, "forceAddress": true, "mtu": 1420, "ipam": { "type": "host-local" } }, { "type": "portmap", "snat": true, "capabilities": { "portMappings": true } } ] } JSON |
5.2.2 CIDR
如下：
| 云主机 | 自带内网 IP | WireGuard VPN IP | Pod CIDR |
| ty1 | 192.168.1.226 | 10.4.0.3/16 | 10.42.0.0/24 |
| ali1 | 172.21.143.136 | 10.4.0.1/16 | 10.42.3.0/24 |
| hw1 | 192.168.7.226 | 10.4.0.4/16 | 10.42.1.0/24 |
| bd1 | 192.168.64.4 | 10.4.0.2/16 | 10.42.4.0/24 |
| bd2 | 192.168.64.5 | 无 | 10.42.5.0/24 |
| tx1 | 172.17.0.3 | 10.4.0.5/16 | 10.42.2.0/24 |
ℹ️ 备注 ：
云主机的自带内网 IP 没有经过任何特殊设置，都是公有云自动生成的。
5.2.3 路由表
再看路由表，以 ty1 为例，如下：
❯ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 100 0 0 eth0 10.4.0.0 0.0.0.0 255.255.0.0 U 0 0 0 kilo0 10.42.0.0 0.0.0.0 255.255.255.0 U 0 0 0 kube-bridge 10.42.1.0 10.4.0.4 255.255.255.0 UG 0 0 0 kilo0 10.42.2.0 10.4.0.5 255.255.255.0 UG 0 0 0 kilo0 10.42.3.0 10.4.0.1 255.255.255.0 UG 0 0 0 kilo0 10.42.4.0 10.4.0.2 255.255.255.0 UG 0 0 0 kilo0 10.42.5.0 10.4.0.2 255.255.255.0 UG 0 0 0 kilo0 172.17.0.3 10.4.0.5 255.255.255.255 UGH 0 0 0 kilo0 172.21.143.136 10.4.0.1 255.255.255.255 UGH 0 0 0 kilo0 192.168.7.226 10.4.0.4 255.255.255.255 UGH 0 0 0 kilo0 192.168.64.4 10.4.0.2 255.255.255.255 UGH 0 0 0 kilo0 192.168.64.5 10.4.0.2 255.255.255.255 UGH 0 0 0 kilo0 BASH |
bd1 的如下：
❯ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface ... 10.42.5.0 192.168.64.5 255.255.255.0 UG 0 0 0 tunl0 ... BASH |
bd2 的如下：
❯ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.64.1 0.0.0.0 UG 100 0 0 eth0 10.4.0.1 192.168.64.4 255.255.255.255 UGH 0 0 0 tunl0 10.4.0.2 192.168.64.4 255.255.255.255 UGH 0 0 0 tunl0 10.4.0.3 192.168.64.4 255.255.255.255 UGH 0 0 0 tunl0 10.4.0.4 192.168.64.4 255.255.255.255 UGH 0 0 0 tunl0 10.4.0.5 192.168.64.4 255.255.255.255 UGH 0 0 0 tunl0 10.42.0.0 192.168.64.4 255.255.255.0 UG 0 0 0 tunl0 10.42.1.0 192.168.64.4 255.255.255.0 UG 0 0 0 tunl0 10.42.2.0 192.168.64.4 255.255.255.0 UG 0 0 0 tunl0 10.42.3.0 192.168.64.4 255.255.255.0 UG 0 0 0 tunl0 10.42.4.0 192.168.64.4 255.255.255.0 UG 0 0 0 tunl0 10.42.5.0 0.0.0.0 255.255.255.0 U 0 0 0 kube-bridge 169.254.169.254 192.168.64.2 255.255.255.255 UGH 100 0 0 eth0 172.17.0.3 192.168.64.4 255.255.255.255 UGH 0 0 0 tunl0 172.21.143.136 192.168.64.4 255.255.255.255 UGH 0 0 0 tunl0 192.168.1.226 192.168.64.4 255.255.255.255 UGH 0 0 0 tunl0 192.168.7.226 192.168.64.4 255.255.255.255 UGH 0 0 0 tunl0 192.168.64.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0 BASH |
5.2.4 总结
- 这台访问
10.4.0.0/16（也就是 WireGuard 网段），都走kilo0网卡（就是 WireGuard 网卡）；
- 云主机间通过 WireGuard VPN IP 互相访问；
- ty1 上 pod 互访（就是访问
10.42.0.0/24网段），走kube-bridge网桥， - ty1 访问其他 Node 的 Pod, 通过
kilo0:
- 访问
10.42.1.0/24(hw1 上的 pod)，SNAT 走 hw1 的 WireGuard VPN IP：10.4.0.4 - 访问
10.42.2.0/24(tx1 上的 pod)，SNAT 走 tx1 的 WireGuard VPN IP：10.4.0.5 - 访问
10.42.3.0/24(ali1 上的 pod)，SNAT 走 ali1 的 WireGuard VPN IP：10.4.0.1 - 访问
10.42.4.0/24(bd1 上的 pod) 和10.42.5.0/24（bd2 上的 pod），SNAT 都走 bd1 的 WireGuard VPN IP：10.4.0.2（因为 bd1 和 bd2 位于同一 VPC 内网，且只有 bd1 上有 WireGuard VPN IP，bd2 上没有）
- ty1 通过云主机自带的内网 IP 访问其他 Node，也是通过
kilo0SNAT 后走对应的 WireGuard VPN IP - bd1 访问
10.42.5.0/24（bd2 上的 pod）, 通过tunl0. - bd2 上，就只有
kube-bridge和tunl0
- 同一台机器 pod 互访，
kube-bridge - 访问其他机器 pod，走
tunl0通过 VPC 内网 IP（192.168.64.4）到 bd，再由 bd1 转发出去。
🗒️ 再凝练下：
- 同一 Node 里，走网桥；
- 同一 location，走自带的内网 IP（VPC 网络）；
- 不同 location，走 WireGuard VPN；
kgctl
Kilo 提供了一个用于检查集群和与集群交互的命令行工具：kgctl。
这个工具可以用来理解一个网格的拓扑结构，获取对等点的 WireGuard 配置，或者绘制一个集群图。
kgctl需要通过设置 KUBECONFIG 环境变量或提供 --kubeconfig 标志来提供 Kubernetes 配置文件。
安装
kgctl二进制文件是为 Linux、macOS 和 Windows 自动编译的，每个版本的 Kilo 都可以从 GitHub 发布页面 下载。
命令
graph
⚠️ 注意 ：
需要安装 circo，否则无法生成拓扑图。
安装命令：sudo apt install circos graphviz -y
graph命令以 GraphViz 格式生成一个表示 Kilo 网格的图形。该图有助于理解或调试网络拓扑。
例子：
kgctl graph SHELL |
这将产生一些 DOT 图描述语言的输出，例如：
digraph kilo { label="10.2.4.0/24"; labelloc=t; outputorder=nodesfirst; overlap=false; "ip-10-0-6-7"->"ip-10-0-6-146"[dir=both]; "ip-10-1-13-74"->"ip-10-1-20-76"[dir=both]; "ip-10-0-6-7"->"ip-10-1-13-74"[dir=both]; "ip-10-0-6-7"->"squat"[dir=both, style=dashed]; "ip-10-1-13-74"->"squat"[dir=both, style=dashed]; # ... } ; DOT |
要渲染图形，使用 GraphViz layout 工具之一，例如circo:
kgctl graph | circo -Tsvg > cluster.svg SHELL |
showconf
showconf命令输出集群中某个节点或对等体的 WireGuard 配置，即该节点或对等体为了参与 mesh 需要在其本地的 WireGuard 接口上设置的配置。例子：
NODE=master # the name of a node kgctl showconf node $NODE SHELL |
这将产生一些 INI 格式的输出，例如。
[Interface] ListenPort = 51820 [Peer] AllowedIPs = 10.2.0.0/24, 10.1.13.74/32, 10.2.4.0/24, 10.1.20.76/32, 10.4.0.2/32 Endpoint = 3.120.246.76:51820 PersistentKeepalive = 0 PublicKey = IgDTEvasUvxisSAmfBKh8ngFmc2leZBvkRwYBhkybUg= INI |
总结
WireGuard 配置
经过 Kilo 自动感知生成的 WireGuard 配置如下：
❯ kgctl showconf node ty1.k3s [Interface] ListenPort = 51820 [Peer] AllowedIPs = 10.42.3.0/24, 172.21.143.136/32, 10.4.0.1/32 Endpoint = [{{ ali1_public_ip }}]:51820 PersistentKeepalive = 20 PublicKey = tscPxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx= [Peer] AllowedIPs = 10.42.4.0/24, 192.168.64.4/32, 10.42.5.0/24, 192.168.64.5/32, 10.4.0.2/32 Endpoint = [{{ bd1_public_ip }}]:51820 PersistentKeepalive = 20 PublicKey = 29khxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxrz8= [Peer] AllowedIPs = 10.42.1.0/24, 192.168.7.226/32, 10.4.0.4/32 Endpoint = [{{ hw1_public_ip }}]:51820 PersistentKeepalive = 20 PublicKey = B9JZe6X8+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx= [Peer] AllowedIPs = 10.42.2.0/24, 172.17.0.3/32, 10.4.0.5/32 Endpoint = [{{ tx1_public_ip }}]:51820 PersistentKeepalive = 20 PublicKey = mn1rUiD+Zb3/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxU= INI |
网络拓扑
如下图：
基于 K3S WireGuard Kilo 的多云统一 K8S 集群网络拓扑
网络流
- 同一 Node 里，走网桥；
- 同一 location，走自带的内网 IP（VPC 网络）；
- 不同 location，走 WireGuard VPN；
🎉🎉🎉
