[TOC]
本文介绍如何在 Ubuntu 22.04 系统上,通过 IntelliJ IDEA 对 containerd 进行源码级调试。我们将从 containerd 的安装、源码编译、验证调试信息的存在,到最终的调试过程中,每一步骤都进行详细讲解。
1 安装 containerd 📦
首先,完成 containerd 的安装
2 源码编译 🔨
源码编译是调试的前提,具体步骤如下:
克隆 containerd 的 GitHub 仓库:🌐
git clone git@github.com:containerd/containerd.git
🌐
切换到目标分支或版本标签,例如切换到
1.7.2
版本:🔀git branch v1.7.2 v1.7.2 && git checkout v1.7.2
下载 containerd 的依赖项:📚
cd containerd && go mod tidy
编译 containerd 源码:🏗️
make build GODEBUG=true all
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# make build GODEBUG=true all
+ build
+ bin/ctr
go build -gcflags=all="-N -l" -gcflags=-trimpath=/root/go/src -buildmode=pie -o bin/ctr -ldflags '-X github.com/containerd/containerd/version.Version=v1.7.2-2-gbe3ad13c1 -X github.com/containerd/containerd/version.Revision=be3ad13c14e0e1da2840fc6496f2bcefefb99764 -X github.com/containerd/containerd/version.Package=github.com/containerd/containerd ' -tags "urfave_cli_no_docs static_build" ./cmd/ctr
+ bin/containerd
go build -gcflags=all="-N -l" -gcflags=-trimpath=/root/go/src -buildmode=pie -o bin/containerd -ldflags '-X github.com/containerd/containerd/version.Version=v1.7.2-2-gbe3ad13c1 -X github.com/containerd/containerd/version.Revision=be3ad13c14e0e1da2840fc6496f2bcefefb99764 -X github.com/containerd/containerd/version.Package=github.com/containerd/containerd ' -tags "urfave_cli_no_docs static_build" ./cmd/containerd
+ bin/containerd-stress
go build -gcflags=all="-N -l" -gcflags=-trimpath=/root/go/src -buildmode=pie -o bin/containerd-stress -ldflags '-X github.com/containerd/containerd/version.Version=v1.7.2-2-gbe3ad13c1 -X github.com/containerd/containerd/version.Revision=be3ad13c14e0e1da2840fc6496f2bcefefb99764 -X github.com/containerd/containerd/version.Package=github.com/containerd/containerd ' -tags "urfave_cli_no_docs static_build" ./cmd/containerd-stress
+ bin/containerd-shim
+ bin/containerd-shim-runc-v1
+ bin/containerd-shim-runc-v2
+ binaries
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# ls -lhtr bin/
total 158M
-rwxr-xr-x 1 root root 31M Jul 29 14:34 ctr
-rwxr-xr-x 1 root root 61M Jul 29 14:34 containerd
-rwxr-xr-x 1 root root 29M Jul 29 14:34 containerd-stress
-rwxr-xr-x 1 root root 9.4M Jul 29 14:34 containerd-shim
-rwxr-xr-x 1 root root 12M Jul 29 14:34 containerd-shim-runc-v1
-rwxr-xr-x 1 root root 17M Jul 29 14:34 containerd-shim-runc-v2
root@containerd:~/workspace/containerd#
注意:编译过程中可能会因为缺少 btrfs 文件系统而报错(
linux/btrfs_tree.h: No such file or directory
)。遵循本文第一步的安装指导可避免此问题。⚠️
3 验证编译的二进制文件是否包含调试信息 🔍
验证编译是否成功并包含调试信息,可以通过以下任一方法:
3.1 使用objdump工具
成功编译的输出示例,会显示含调试信息的输出objdump --syms bin/containerd
✅
root@containerd:~/workspace/containerd# objdump --syms bin/containerd
bin/containerd: file format elf64-x86-64
SYMBOL TABLE:
0000000000000000 l df *ABS* 0000000000000000 Scrt1.o
00000000000003b4 l O .note.ABI-tag 0000000000000020 __abi_tag
0000000000000000 l df *ABS* 0000000000000000 go.go
00000000003ad4c0 l F .text 0000000000000000 runtime.text
00000000003ad4c0 l F .text 0000000000000059 internal/cpu.Initialize
00000000003ad520 l F .text 0000000000000537 internal/cpu.processOptions
00000000003ada60 l F .text 0000000000000026 internal/cpu.indexByte
00000000003adaa0 l F .text 0000000000000925 internal/cpu.doinit
00000000003ae3e0 l F .text 0000000000000006 internal/cpu.isSet
00000000003ae400 l F .text 000000000000001b internal/cpu.cpuid.abi0
00000000003ae420 l F .text 0000000000000011 internal/cpu.xgetbv.abi0
00000000003ae440 l F .text 0000000000000009 internal/cpu.getGOAMD64level.abi0
00000000003ae460 l F .text 000000000000007a type:.eq.internal/cpu.option
00000000003ae4e0 l F .text 00000000000000e6 type:.eq.[6]internal/cpu.option
00000000003ae5e0 l F .text 0000000000000003 runtime/internal/atomic.(*Int32).Load
00000000003ae600 l F .text 0000000000000003 runtime/internal/atomic.(*Int32).Store
00000000003ae620 l F .text 000000000000000d runtime/internal/atomic.(*Int32).CompareAndSwap
00000000003ae640 l F .text 000000000000000a runtime/internal/atomic.(*Int32).Add
00000000003ae660 l F .text 0000000000000004 runtime/internal/atomic.(*Int64).Load
00000000003ae680 l F .text 0000000000000004 runtime/internal/atomic.(*Int64).Store
00000000003ae6a0 l F .text 000000000000000f runtime/internal/atomic.(*Int64).CompareAndSwap
00000000003ae6c0 l F .text 0000000000000007 runtime/internal/atomic.(*Int64).Swap
00000000003ae6e0 l F .text 000000000000000d runtime/internal/atomic.(*Int64).Add
00000000003ae700 l F .text 0000000000000003 runtime/internal/atomic.(*Uint8).Load
00000000003ae720 l F .text 0000000000000003 runtime/internal/atomic.(*Uint8).Store
00000000003ae740 l F .text 0000000000000004 runtime/internal/atomic.(*Uint8).And
00000000003ae760 l F .text 0000000000000004 runtime/internal/atomic.(*Uint8).Or
00000000003ae780 l F .text 0000000000000024 runtime/internal/atomic.(*Bool).Load
00000000003ae7c0 l F .text 000000000000001f runtime/internal/atomic.(*Bool).Store
00000000003ae7e0 l F .text 0000000000000003 runtime/internal/atomic.(*Uint32).Load
00000000003ae800 l F .text 0000000000000003 runtime/internal/atomic.(*Uint32).LoadAcquire
00000000003ae820 l F .text 0000000000000003 runtime/internal/atomic.(*Uint32).Store
00000000003ae840 l F .text 0000000000000003 runtime/internal/atomic.(*Uint32).StoreRelease
00000000003ae860 l F .text 000000000000000d runtime/internal/atomic.(*Uint32).CompareAndSwap
00000000003ae880 l F .text 000000000000000d runtime/internal/atomic.(*Uint32).CompareAndSwapRelease
00000000003ae8a0 l F .text 0000000000000005 runtime/internal/atomic.(*Uint32).Swap
00000000003ae8c0 l F .text 0000000000000004 runtime/internal/atomic.(*Uint32).And
00000000003ae8e0 l F .text 0000000000000004 runtime/internal/atomic.(*Uint32).Or
若输出提示 no symbols
,则表示该二进制文件不包含调试信息。❌
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# objdump --syms /usr/local/bin/containerd
/usr/local/bin/containerd: file format elf64-x86-64
SYMBOL TABLE:
no symbols
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#
3.2 使用file工具
含调试信息的文件将显示 with debug_info, not stripped
。✅
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# file bin/containerd
bin/containerd: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=9bb291e61e1eceb23359dc29100845e5c1edf763, for GNU/Linux 3.2.0, with debug_info, not stripped
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#
若显示 stripped
,则不包含调试信息。❌
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# file /usr/local/bin/containerd
/usr/local/bin/containerd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=322f89b7e351fe2ccfaa0fe30de79c76d49d6e26, for GNU/Linux 3.2.0, stripped
root@containerd:~/workspace/containerd#
3.3 dlv工具验证
正确的调试信息如下所示:✅
root@containerd:~/workspace/containerd# dlv exec bin/containerd
Type 'help' for list of commands.
(dlv)
(dlv)
(dlv)
(dlv)
(dlv)
(dlv) exit
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#
错误的调试信息如下,会提示:no debug info found
,如下所示:❌
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# dlv exec /usr/local/bin/containerd
Warning: no debug info found, some functionality will be missing such as stack traces and variable evaluation.
Type 'help' for list of commands.
(dlv)
(dlv)
(dlv)
(dlv)
(dlv) exit
4 debug
debug containerd
步骤如下:
1、利用
ps -ef|gerp containerd
查看containerd
启动所需要的参数,这一步特别重要,尤其是在debug k8s
源码的时候,k8s
的每一个组件都带了很多命令行参数,想要调试这些组件,必须把这些组件的启动参数原封不动的加入到dlv
调试命令当中- 注意,实际上执行上述命令之后会发现,
containerd
并没有启动参数,因此无需关心。
- 注意,实际上执行上述命令之后会发现,
2、通过
dlv
命令启动containerd
,启动命令我们可以从IDEA remote debug
功能拷贝过来🔍dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd
📡注意:这里在启动
contaienrd
的时候没有指定任何参数,实际上也可以根据自己的情况加入containerd
参数,譬如指定containerd
的配置文件的位置(containerd
默认配置文件为:/etc/containerd/config.toml
),也可以指定调试的debug
级别,譬如:dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --config=/root/mycontainerd/config.toml --log-level=debug
dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --log-level=debug
3、在
IDEA
启动debug
,连接到远程调试📍- 注意:在启动
IDEA
调试之前,你需要在想要debug
的位置增加断点,否则程序启动会直接运行起来,等你这个时候打断点,很可能就晚了。 - 注意:在启动
IDEA
调试之前,你需要修改IDEA
操作系统标识为Linux
- 注意:在启动
执行dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --log-level=debug
命令之后,此时会阻塞在这里,千万不要使用ctrl + c
,只有当IDEA
连接上来的时候才会开始执行🚫⌛
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --log-level=debug
API server listening at: [::]:12345
2023-07-29T15:06:45+08:00 warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)