总体架构
整个K8S集群由2台无影ubuntu云桌面组成，分别部署master节点和worker节点
通过无影web客户端连接对应的云桌面。
云桌面内安装vscode和go调试工具dlv，并且可以访问互联网。
建立K8S集群
检查集群
集群默认是搭建完成正常运行的，在master节点上执行命令, 并检查输出结果，如下图所示。
K8S集群有2个工作节点，1个是master， ROLE为conrol-plane，1个是worker node， ROLE为none。
kube-system包含的apiserver/controller/scheduler/etcd/proxy/dns等组件运行正常。
flannel网络插件运行正常
手动配置集群
若命令执行，结果不符合上图，可以手动恢复集群。
初始化脚本默认在 /opt/wuying/myspace/k8s 下面
在master节点上，用 reset-k8s-master.sh 重新初始化集群, 然后执行 config-flannel.sh 配置网络插件，最后执行 kubeadm token create --print-join-command ，或者worker节点加入集群的命令，如下图所示
在worker节点上，用 reset-k8s-worker.sh 重置初始化集群，然后用上面在master节点上执行 kubeadm token create --print-join-command命令的结果，重新 join集群
完成后重新检查集群的状态。
ide-vscode
参考 https://code.visualstudio.com/docs/editor/debugging
https://github.com/golang/vscode-go/blob/master/docs/debugging.md
调试工具-dlv
参考 https://github.com/go-delve/delve
编译代码
以kube-apiserver举例
#设置kubernetes源码版本
cd /go/src/k8s.io/kubernetes
git checkout v1.25.2
#编译kube-apiserver,编译的apiserver在当前目录下，该目录后续在编译调试的时候需要
#编译所有master组件
make DBG=1
# 单独编译kube-apiserver组件
make DBG=1 WHAT="cmd/kube-apiserver"
代码目录结构
代码地址: https://github.com/kubernetes/kubernetes
调试步骤（dlv exec）
停止组件运行
#查找kube-apiserver运行参数
ps|grep apiserver
kube-apiserver --advertise-address=192.168.30.87 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
#停止kube-apiserver的运行
sudo crictl ps|grep apiserver
sudo crictl stop ${apiserver-container-id}
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml ~/
#检查kube-apiserver是否还在运行
ps -elf | grep kube-apiserver
运行自己编译的组件
#使用dlv exec执行自己编译的kube-apiserver
sudo /go/bin/dlv exec /go/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver --headless -l :2345 --api-version=2 --accept-multiclient -- --advertise-address=192.168.30.87 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
执行结果
vscode调试源码
打开kubernetes所在目录:/go/src/k8s.io/kubernetes
设置断点
打开cmd/kube-apiserver/apiserver.go文件，并设置断点
配置dlv server参数
在 vscode 上点击 Debug，点击 create a launch.json file. 选择Go: connect to server，之后全部默认设置，最后生成的launch.json如下图所示
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Connect to server",
"type": "go",
"request": "attach",
"mode": "remote",
"remotePath": "${workspaceFolder}",
"port": 2345,
"host": "127.0.0.1"
}
]
}
连接dlv server
选择connect to server并点击绿色的三角形进行debug
查询断点处
可以看到如下所示，程序运行到断点处
断开dlv server后重连处理
当点击断开连接后，需要重新用dlv运行kube-apiserver
#kill掉运行中的kube-apiserver
ps -elf | grep apiserver
sudo kill <pid of apiserver>
#重新运行kube-apiserver
sudo sudo /go/bin/dlv exec --headless -l :2345 --api-version=2 --accept-multiclient /go/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver -- --advertise-address=192.168.30.87 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
调试步骤（dlv attach）
停止组件运行（同exec）
运行自己编译的组件
#运行自己编译的Kube-apiserver
sudo /go/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver --advertise-address=192.168.30.87 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
#查询kube-apiserver的pid
ps -elf | grep kube-apiserver
查询组件进程id
dlv接入组件进程
#使用dlv attach kube-apiserver的pid
sudo /go/bin/dlv attach 203315 --headless --listen=:2345 --api-version=2 --accept-multiclient --log
执行结果
vscode调试源码
打开kubernetes所在目录:/go/src/k8s.io/kubernetes
配置dlv server参数
在 vscode 上点击 Debug，点击 create a launch.json file. 选择Go: connect to server，之后全部默认设置，最后生成的launch.json如下图所示
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Connect to server",
"type": "go",
"request": "attach",
"mode": "remote",
"remotePath": "${workspaceFolder}",
"port": 2345,
"host": "127.0.0.1"
}
]
}
连接dlv server
选择connect to server并点击绿色的三角形进行debug
设置断点
以kubectl get pods举例设置断点如下
vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go GetResource函数处
查询断点处
执行kubectl get pod会发现程序停止到我们设置的获取pods的函数处
