什么是kubeshark
Kubeshark 由 2021 年 UP9 公司开源的 K8s API 流量查看器 Mizu 发展而来,试图成为一款 K8s 全过程流量监控工具。
Kubeshark 被叫做 kubernetes 的 API 流量查看器,它提供对进出 Kubernetes 集群内容器和 pod 的所有 API 流量和负载的深度可见性和监控。类似于针对 Kubernetes 而重新发明的 TCPDump 和 Wireshark。地址:https://github.com/kubeshark/kubeshark
Kubeshark 也被叫做 Kubernetes 的可观测性工具,可以对微服务进行动态分析,检测异常并在运行时出现某些模式时触发功能。
- 1.可以将 Kubeshark 视为 Wireshark、BPF 编译器集合 (BCC) 工具等的 Kubernetes 感知组合。
- 2.Kubeshark 可以嗅探集群中的部分或所有 TCP 流量,将其记录到 PCAP 文件中并剖析。
- 3.Kubeshark 使用 eBPF 来跟踪内核空间和用户空间中的函数调用。
安装kubeshark
安装之前需要对其架构有些了解
一共由三部分组成分别是:
- 1.CLI,它是客户端的 二进制文件,通过 K8s API 与集群通信。
- 2.Hub,它协调 worker 部署,接收来自每个 worker 的嗅探和剖析,并收集到一个中心位置。它还提供一个Web界面,用于在浏览器上显示收集到的流量。
- 3.Work,作为 DaemonSet 部署到集群中,以确保集群中的每个节点都被 Kubeshark 覆盖。
启动后三个组件的运行情况,一定要对其限制资源,否则可能会拖垮集群。
启动kubeshark
监控所有namespace的pod流量
CLI启动
$ ./kubeshark.exe --docker-registry 10.50.10.185/kubeshark/docker.io/kubeshark/ --debug tap -A 2023-04-07T08:56:20+08:00 DBG config.go:79 > Init config is finished. config={"Config":{"Regenerate":false},"DumpLogs":false,"HeadlessMode":false,"Kube":{"ConfigPathStr":"","Context":""},"License":"","Logs":{"FileStr":""},"ResourceLabels":{},"Scripting":{"Env":null,"Source":"","WatchScripts":true},"Tap":{"AllNamespaces":true,"Debug":true,"Docker":{"ImagePullPolicy":"Always","ImagePullSecrets":null,"Registry":"10.50.10.185/kubeshark/docker.io/kubeshark/","Tag":"latest"},"DryRun":false,"Namespaces":null,"PacketCapture":"libpcap","Pcap":"","PodRegexStr":".*","Proxy":{"Front":{"DstPort":80,"SrcPort":8899},"Host":"127.0.0.1","Hub":{"DstPort":80,"SrcPort":8898},"Worker":{"DstPort":8897,"SrcPort":8897}},"Resources":{"Hub":{"CpuLimit":"750m","CpuRequests":"50m","MemoryLimit":"1Gi","MemoryRequests":"50Mi"},"Worker":{"CpuLimit":"750m","CpuRequests":"50m","MemoryLimit":"1Gi","MemoryRequests":"50Mi"}},"SelfNamespace":"kubeshark","ServiceMesh":true,"StorageLimit":"200MB","Tls":true}} 2023-04-07T08:56:20+08:00 INF tapRunner.go:53 > Using Docker: registry=10.50.10.185/kubeshark/docker.io/kubeshark/ tag=latest 2023-04-07T08:56:20+08:00 INF tapRunner.go:61 > Kubeshark will store the traffic up to a limit (per node). Oldest TCP streams will be removed once the limit is reached. limit=200MB 2023-04-07T08:56:20+08:00 INF versionCheck.go:23 > Checking for a newer version... 2023-04-07T08:56:20+08:00 DBG provider.go:83 > K8s client config. api-path= host=https://10.50.10.108:6443 user-agent= 2023-04-07T08:56:20+08:00 INF common.go:69 > Using kubeconfig: path="C:\\Users\\00625\\.kube\\config" 2023-04-07T08:56:20+08:00 INF tapRunner.go:82 > Targeting pods in: namespaces=[""] 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: busybox 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: dnsutils 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: myapp-6c84fdb964-q9jgg 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: myapp-6c84fdb964-zx8sq 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: nfs-deploy-187-sc-77d958d57d-7k82m 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: nginx-web-2ztjg 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: nginx-web-jl78g 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: rpt-etl-loadmesdefectdata-meqmsgp-final-deployment-54b557fzzlzc 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: tea-797c7b5d89-26clj 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: tea-797c7b5d89-h7m6j 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: tea-797c7b5d89-xv8qq 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: configmap-demo-pod 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: etl-loadedcdata-oc-test-deployment-765bdcbb57-jqr96 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: nginx-deployment-776bf658dc-5qmwv 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: nginx-deployment-776bf658dc-dpl77 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: nginx-deployment-776bf658dc-vgwmq 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: nginx-web-bcrvx 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: nginx-web-tb9b9 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: pc-deployment-5dbbbc49b9-pdnkn 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: pc-deployment-5dbbbc49b9-sjthh 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: pod-configmap 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: pod-liveness-httpget 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: qms-deploy-7f5ccdcc8b-dhn6g 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: tomcat-deployment-869c6bb5d-b9vgp 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: tomcat-deployment-869c6bb5d-ckptc 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: tomcat-deployment-869c6bb5d-ngd2k 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: web-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: web-1 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: cloud-eureka-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: db-example-7f5b967b5b-hcl7b 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: gateway-example-97868d74-mpxwl 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: svc-example-57c4bcd684-nsrrg 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: web-example-74cf6f548c-njlnm 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-gitlab-runner-minio-gitlab-runner-75f87cfdbf-d89z6 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: gitlab-8d977cc89-bj76w 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: gitlib-db-postgresql-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: gitlib-redis-node-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: ingress-nginx-controller-7466f577fb-kd6r8 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: zentao-9d66c8d6c-4fkwv 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kafka-client 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kafka-cluster-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: zookeeper-cluster-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: calico-kube-controllers-7ff9f9d7ff-xw775 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: calico-node-45t6m 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: calico-node-8qxkq 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: calico-node-9twcw 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: calico-node-bfntl 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: calico-node-dwxj9 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: calico-node-ms4pr 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: coredns-66dbf4b6cd-2zk95 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: coredns-66dbf4b6cd-qlzbk 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: coredns-66dbf4b6cd-qmjcn 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: eip-nfs-chot-nas-500gb-6bbfddcc9b-mt9jj 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: dashboard-metrics-scraper-549b9db55d-wc7m8 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kubernetes-dashboard-6c57ff6878-45czg 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kuboard-loki-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kuboard-loki-grafana-54fc8c475-qrh2g 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kuboard-promtail-b8clp 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kuboard-promtail-dkj42 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kuboard-promtail-gl9xm 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kuboard-promtail-h7m8h 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kuboard-promtail-znr8g 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kuboard-pv-browser-bmrd4 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kuboard-pv-browser-fxlfz 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kuboard-pv-browser-g8fw5 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kuboard-pv-browser-pnxjb 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: kuboard-pv-browser-vpggp 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-minio-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-minio-1 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-minio-2 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-minio-3 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: mysql-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-nexus-nexus-repository-manager-6595d7c79b-q7znf 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: nfs-client-provisioner-sc179-8447d45564-tfxfd 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: alertmanager-chot-prometheus-kube-prome-alertmanager-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-prometheus-grafana-7fccf797f5-jrqsx 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-prometheus-kube-prome-operator-5754f74d75-lfngs 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-prometheus-kube-state-metrics-89975d948-r6dx6 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: prometheus-chot-prometheus-kube-prome-prometheus-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-rabbitmq-dev-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-rabbitmq-dev-1 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-rabbitmq-0 2023-04-07T08:56:21+08:00 INF tapRunner.go:142 > Targeted pod: chot-rabbitmq-1 2023-04-07T08:56:21+08:00 INF tapRunner.go:92 > Waiting for the creation of Kubeshark resources... 2023-04-07T08:56:21+08:00 INF createResources.go:111 > Successfully created a pod. pod=kubeshark-hub 2023-04-07T08:56:21+08:00 INF createResources.go:123 > Successfully created a pod. pod=kubeshark-front 2023-04-07T08:56:21+08:00 INF createResources.go:72 > Successfully created a service. service=kubeshark-hub 2023-04-07T08:56:21+08:00 INF createResources.go:79 > Successfully created a service. service=kubeshark-front 2023-04-07T08:56:21+08:00 DBG wait.go:13 > Waiting to finish... 2023-04-07T08:56:21+08:00 INF tapRunner.go:264 > Added: pod=kubeshark-front 2023-04-07T08:56:21+08:00 INF tapRunner.go:173 > Added: pod=kubeshark-hub 2023-04-07T08:56:21+08:00 DBG tapRunner.go:281 > Watching pod. containers-statuses=[{"image":"10.50.10.185/kubeshark/docker.io/kubeshark/front:latest","imageID":"","lastState":{},"name":"kubeshark-front","ready":false,"restartCount":0,"started":false,"state":{"waiting":{"reason":"ContainerCreating"}}}] phase=Pending pod=kubeshark-front 2023-04-07T08:56:21+08:00 DBG versionCheck.go:38 > Fetched the latest release: elapsed-time=435.195 local-version=39.5 upstream-version=39.5 2023-04-07T08:56:22+08:00 DBG tapRunner.go:281 > Watching pod. containers-statuses=[{"image":"10.50.10.185/kubeshark/docker.io/kubeshark/front:latest","imageID":"","lastState":{},"name":"kubeshark-front","ready":false,"restartCount":0,"started":false,"state":{"waiting":{"reason":"ContainerCreating"}}}] phase=Pending pod=kubeshark-front 2023-04-07T08:56:22+08:00 DBG tapRunner.go:190 > Watching pod. containers-statuses=[{"image":"10.50.10.185/kubeshark/docker.io/kubeshark/hub:latest","imageID":"","lastState":{},"name":"kubeshark-hub","ready":false,"restartCount":0,"started":false,"state":{"waiting":{"reason":"ContainerCreating"}}}] phase=Pending pod=kubeshark-hub 2023-04-07T08:56:22+08:00 DBG tapRunner.go:369 > Watching events. event=kubeshark-hub.1753810db14a1df9 kind=Pod name=kubeshark-hub note="Pulling image \"10.50.10.185/kubeshark/docker.io/kubeshark/hub:latest\"" pod=kubeshark-hub reason=Pulling 2023-04-07T08:56:22+08:00 DBG tapRunner.go:369 > Watching events. event=kubeshark-hub.1753810dbb96fbca kind=Pod name=kubeshark-hub note="Successfully pulled image \"10.50.10.185/kubeshark/docker.io/kubeshark/hub:latest\" in 172.792597ms" pod=kubeshark-hub reason=Pulled 2023-04-07T08:56:22+08:00 DBG tapRunner.go:369 > Watching events. event=kubeshark-hub.1753810dbd12f28b kind=Pod name=kubeshark-hub note="Created container kubeshark-hub" pod=kubeshark-hub reason=Created 2023-04-07T08:56:22+08:00 DBG tapRunner.go:369 > Watching events. event=kubeshark-hub.1753810dc34b23fe kind=Pod name=kubeshark-hub note="Started container kubeshark-hub" pod=kubeshark-hub reason=Started 2023-04-07T08:56:23+08:00 DBG tapRunner.go:281 > Watching pod. containers-statuses=[{"containerID":"docker://05b0472bc9b3ebbed8df9c299a48b04221bfba87e09e88a2f5374e86db887d40","image":"10.50.10.185/kubeshark/docker.io/kubeshark/front:latest","imageID":"docker-pullable://10.50.10.185/kubeshark/docker.io/kubeshark/front@sha256:80a9dee0a3e473d960f90230862767012ba3b32f8ee13b322ee59f35b3eddbb0","lastState":{},"name":"kubeshark-front","ready":false,"restartCount":0,"started":true,"state":{"running":{"startedAt":"2023-04-07T00:56:21Z"}}}] phase=Running pod=kubeshark-front 2023-04-07T08:56:23+08:00 DBG tapRunner.go:281 > Watching pod. containers-statuses=[{"containerID":"docker://05b0472bc9b3ebbed8df9c299a48b04221bfba87e09e88a2f5374e86db887d40","image":"10.50.10.185/kubeshark/docker.io/kubeshark/front:latest","imageID":"docker-pullable://10.50.10.185/kubeshark/docker.io/kubeshark/front@sha256:80a9dee0a3e473d960f90230862767012ba3b32f8ee13b322ee59f35b3eddbb0","lastState":{},"name":"kubeshark-front","ready":true,"restartCount":0,"started":true,"state":{"running":{"startedAt":"2023-04-07T00:56:21Z"}}}] phase=Running pod=kubeshark-front 2023-04-07T08:56:23+08:00 DBG tapRunner.go:190 > Watching pod. containers-statuses=[{"containerID":"docker://5872d7590936c6b43c2b06cd47a027651e6b6ae14e61d488bb8af4bd40b6bc3d","image":"10.50.10.185/kubeshark/docker.io/kubeshark/hub:latest","imageID":"docker-pullable://10.50.10.185/kubeshark/docker.io/kubeshark/hub@sha256:de70ee57cd66b6225e43d8500ef95e4f8ef2cd5e8ad7646a11220c87d83ed145","lastState":{},"name":"kubeshark-hub","ready":true,"restartCount":0,"started":true,"state":{"running":{"startedAt":"2023-04-07T00:56:22Z"}}}] phase=Running pod=kubeshark-hub 2023-04-07T08:56:23+08:00 INF proxy.go:29 > Starting proxy... namespace=kubeshark service=kubeshark-hub src-port=8898 2023-04-07T08:56:23+08:00 DBG hub.go:45 > Connection test passed successfully. url=http://localhost:8898 2023-04-07T08:56:23+08:00 INF workers.go:33 > Creating the worker DaemonSet... 2023-04-07T08:56:23+08:00 DBG provider.go:648 > Applying worker DaemonSets. daemonset-name=kubeshark-worker-daemon-set image=10.50.10.185/kubeshark/docker.io/kubeshark/worker:latest namespace=kubeshark pod=kubeshark-worker 2023-04-07T08:56:23+08:00 INF workers.go:52 > Successfully created the worker DaemonSet. 2023-04-07T08:56:23+08:00 INF tapRunner.go:436 > Waiting for worker containers... 2023-04-07T08:56:29+08:00 DBG hub.go:112 > Reported storage limit to Hub: limit=200000000 2023-04-07T08:56:29+08:00 DBG hub.go:145 > Reported pod regex to Hub: namespaces=[""] regex=.* 2023-04-07T08:56:29+08:00 DBG hub.go:344 > Reported POST scripts done to Hub. 2023-04-07T08:56:29+08:00 INF tapRunner.go:471 > Hub is available at: url=http://localhost:8898 2023-04-07T08:56:29+08:00 INF proxy.go:29 > Starting proxy... namespace=kubeshark service=kubeshark-front src-port=8899 2023-04-07T08:56:29+08:00 DBG hub.go:45 > Connection test passed successfully. url=http://localhost:8899 2023-04-07T08:56:29+08:00 INF tapRunner.go:492 > Kubeshark is available at: url=http://localhost:8899
前台展示
过滤出失败的API 调用
http and response.status >= 400
kubeshark-serviceMesh
部署完成后,Kubeshark CLI 将在 http://localhost:8899 打开 UI 单击右上角名为 Service Map 的按钮打开服务依赖关系图。该图根据网络流量显示 Pod 以及它们之间的关系。