引言
最近听好多朋友说，cilium很强，势必将成为主流。因其使用了ebpf，性能好，而且支持网络策略。于是，决定花点时间学习一下。在通过官网文档学习过程中，发现使用cilium作为CNI，居然可以不用安装kube-proxy了。这让我想起来，之前在面试中被问到的一个问题，面试官问我:kube-proxy是否可以不用安装，是否有其他替代品。这下不就有答案了嘛。
顺便吐槽一下，看官方文档学习，是真的有点难(毕竟全英文)；不过还是建议大家看官方文档学习，不要翻译成中文哦。那么接下来，就由我来实操一下。
环境说明
| 序号 | 事项 | 说明 |
| 1 | kubernetes version | v1.21.3 |
| 2 | cilium version | v1.10.3 |
| 3 | kubernetes安装方式 | kubeadm |
| 4 | cilium组网模式 | vxlan |
| 5 | os | ubuntu 18.04 |
| 6 | kubernetes集群规模 | 1master、2node |
正文
- 在master上初始化集群，并通过添加--skip-phases=addon/kube-proxy参数忽略kube-proxy的安装
kubeadm init --apiserver-advertise-address=10.211.55.50 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.21.3 --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all --skip-phases=addon/kube-proxy
- 在两个node上执行kubeadm join，加入集群
kubeadm join 10.211.55.50:6443 --token ouez6j.02ms269v8i4psl7p --discovery-token-ca-cert-hash sha256:5fdafe0fe1adb3b60cd7bc33f033f028279a94a3944816424cc7f5bb498f6868
- 使用helm(v3)来安装cilium。先添加cilium库
helm repo add cilium https://helm.cilium.io/
- 使用如下命令安装cilium，添加kubeProxyReplacement=strict参数
helm install cilium cilium/cilium --version 1.10.3 --namespace kube-system --set kubeProxyReplacement=strict --set k8sServiceHost=10.211.55.50 --set k8sServicePort=6443
- 检查cilium安装结果
# 查看cilium agent,以daemonset方式部署在每个node节点上 root@cilium1:/# kubectl -n kube-system get pods -l k8s-app=cilium NAME READY STATUS RESTARTS AGE cilium-8gwg2 1/1 Running 0 8m4s cilium-t9ffc 1/1 Running 0 8m39s cilium-x42r6 1/1 Running 0 8m16s # 查看cilum operator root@cilium1:~# kubectl get po -A -o wide |grep cilium-operator kube-system cilium-operator-5df88875-867hd 1/1 Running 5 41h 172.16.88.47 cilium3 <none> <none> kube-system cilium-operator-5df88875-9kx8c 1/1 Running 5 41h 172.16.88.253 cilium2 <none> <none>
- 检查是否有kube-proxy组件。可以发现并没有该组件
root@cilium1:/# kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE cilium-8gwg2 1/1 Running 0 10m cilium-operator-5df88875-867hd 1/1 Running 5 27h cilium-operator-5df88875-9kx8c 1/1 Running 5 27h cilium-t9ffc 1/1 Running 0 11m cilium-x42r6 1/1 Running 0 10m coredns-59d64cd4d4-hbwg4 1/1 Running 1 27h coredns-59d64cd4d4-l2pmt 1/1 Running 1 27h etcd-cilium1 1/1 Running 2 27h kube-apiserver-cilium1 1/1 Running 2 27h kube-controller-manager-cilium1 1/1 Running 2 27h kube-scheduler-cilium1 1/1 Running 2 27h
- 检查cilium状态，确保安装正确
root@cilium1:/# kubectl exec -n kube-system cilium-t9ffc -- cilium status Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), clean-cilium-state (init) KVStore: Ok Disabled Kubernetes: Ok 1.21 (v1.21.3) [linux/amd64] Kubernetes APIs: ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"] KubeProxyReplacement: Strict [eth0 10.211.55.50 (Direct Routing)] Cilium: Ok 1.10.3 (v1.10.3-4145278) NodeMonitor: Listening for events on 8 CPUs with 64x4096 of shared memory Cilium health daemon: Ok IPAM: IPv4: 2/254 allocated from 10.0.0.0/24, BandwidthManager: Disabled Host Routing: Legacy Masquerading: BPF [eth0] 10.0.0.0/24 [IPv4: Enabled, IPv6: Disabled] Controller Status: 20/20 healthy Proxy Status: OK, ip 10.0.0.41, 0 redirects active on ports 10000-20000 Hubble: Ok Current/Max Flows: 817/4095 (19.95%), Flows/s: 0.95 Metrics: Disabled Encryption: Disabled Cluster health: 3/3 reachable (2021-08-07T15:29:05Z)
- 部署nginx来测试一下网络联通性
# nginx deployment yaml文件 cat deployment-nginx.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: selector: matchLabels: run: nginx replicas: 4 template: metadata: labels: run: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 # 创建nginx deployment kubectl create -f deployment-nginx.yaml # 查看部署结果 root@cilium1:/# kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-649c4b9857-8f2v5 1/1 Running 1 26h 10.0.2.212 cilium2 <none> <none> nginx-649c4b9857-mhsxs 1/1 Running 1 26h 10.0.1.23 cilium3 <none> <none> nginx-649c4b9857-qw2jj 1/1 Running 1 26h 10.0.2.69 cilium2 <none> <none> nginx-649c4b9857-vj9w2 1/1 Running 1 26h 10.0.1.126 cilium3 <none> <none>
- 创建一个nodeport service来验证service的可访问
# 创建service kubectl expose deployment nginx --type=NodePort --port=80 # 查看service root@cilium1:/# kubectl get svc nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx NodePort 10.97.209.103 <none> 80:31126/TCP 26h
- 验证nodeport、cluster可访问
# 通过nodeport root@cilium1:/# curl 127.0.0.1:31126 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> # 通过service:port root@cilium1:/# curl 10.97.209.103 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> # 检查iptables 发现为空 root@cilium1:/# iptables-save | grep KUBE-SVC root@cilium1:/# # 检查ciliun service root@cilium1:/# kubectl exec -n kube-system cilium-t9ffc -- cilium service list Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), clean-cilium-state (init) ID Frontend Service Type Backend 1 10.96.0.1:443 ClusterIP 1 => 172.16.88.57:6443 2 10.96.0.10:9153 ClusterIP 1 => 10.0.2.229:9153 2 => 10.0.2.80:9153 3 10.96.0.10:53 ClusterIP 1 => 10.0.2.229:53 2 => 10.0.2.80:53 4 10.97.209.103:80 ClusterIP 1 => 10.0.2.69:80 2 => 10.0.1.23:80 3 => 10.0.1.126:80 4 => 10.0.2.212:80 5 172.16.88.57:31126 NodePort 1 => 10.0.2.69:80 2 => 10.0.1.23:80 3 => 10.0.1.126:80 4 => 10.0.2.212:80 6 0.0.0.0:31126 NodePort 1 => 10.0.2.69:80 2 => 10.0.1.23:80 3 => 10.0.1.126:80 4 => 10.0.2.212:80
- 从上面的安装和测试结果来，虽然我们没有安装k8s的kube-proxy组件，但是集群依然正常。说明kube-proxy组件确实是可以被替代的。
总结
以上虽然完成了kubernetes without kube-proxy的搭建和测试工作，但还是有很多事情没说明。比如使用cilium的系统要求、cilium是什么、有几种组网模式、网络策略。不过请不要着急，期待我后续的文章。
