通过 IntelliJ IDEA 对 containerd 进行源码级调试

[TOC]

本文介绍如何在 Ubuntu 22.04 系统上，通过 IntelliJ IDEA 对 containerd 进行源码级调试。我们将从 containerd 的安装、源码编译、验证调试信息的存在，到最终的调试过程中，每一步骤都进行详细讲解。

1 安装 containerd 📦

首先，按照以下链接中的指引完成 containerd 的安装过程：Ubuntu 22.04 安装 containerd 教程 👈

2 源码编译 🔨

源码编译是调试的前提，具体步骤如下：

1. 克隆 containerd 的 GitHub 仓库：🌐

   • git clone git@github.com:containerd/containerd.git 🌐

2. 切换到目标分支或版本标签，例如切换到 1.7.2 版本：🔀

   • git branch v1.7.2 v1.7.2 && git checkout v1.7.2

3. 下载 containerd 的依赖项：📚

   • cd containerd && go mod tidy

4. 编译 containerd 源码：🏗️

   • make build GODEBUG=true all

root@containerd:~/workspace/containerd# 
root@containerd:~/workspace/containerd# 
root@containerd:~/workspace/containerd# 
root@containerd:~/workspace/containerd# make build GODEBUG=true all
+ build
+ bin/ctr
go build -gcflags=all="-N -l" -gcflags=-trimpath=/root/go/src -buildmode=pie  -o bin/ctr -ldflags '-X github.com/containerd/containerd/version.Version=v1.7.2-2-gbe3ad13c1 -X github.com/containerd/containerd/version.Revision=be3ad13c14e0e1da2840fc6496f2bcefefb99764 -X github.com/containerd/containerd/version.Package=github.com/containerd/containerd  ' -tags "urfave_cli_no_docs static_build"  ./cmd/ctr
+ bin/containerd
go build -gcflags=all="-N -l" -gcflags=-trimpath=/root/go/src -buildmode=pie  -o bin/containerd -ldflags '-X github.com/containerd/containerd/version.Version=v1.7.2-2-gbe3ad13c1 -X github.com/containerd/containerd/version.Revision=be3ad13c14e0e1da2840fc6496f2bcefefb99764 -X github.com/containerd/containerd/version.Package=github.com/containerd/containerd  ' -tags "urfave_cli_no_docs static_build"  ./cmd/containerd
+ bin/containerd-stress
go build -gcflags=all="-N -l" -gcflags=-trimpath=/root/go/src -buildmode=pie  -o bin/containerd-stress -ldflags '-X github.com/containerd/containerd/version.Version=v1.7.2-2-gbe3ad13c1 -X github.com/containerd/containerd/version.Revision=be3ad13c14e0e1da2840fc6496f2bcefefb99764 -X github.com/containerd/containerd/version.Package=github.com/containerd/containerd  ' -tags "urfave_cli_no_docs static_build"  ./cmd/containerd-stress
+ bin/containerd-shim
+ bin/containerd-shim-runc-v1
+ bin/containerd-shim-runc-v2
+ binaries
root@containerd:~/workspace/containerd# 
root@containerd:~/workspace/containerd# 
root@containerd:~/workspace/containerd# ls -lhtr bin/
total 158M
-rwxr-xr-x 1 root root  31M Jul 29 14:34 ctr
-rwxr-xr-x 1 root root  61M Jul 29 14:34 containerd
-rwxr-xr-x 1 root root  29M Jul 29 14:34 containerd-stress
-rwxr-xr-x 1 root root 9.4M Jul 29 14:34 containerd-shim
-rwxr-xr-x 1 root root  12M Jul 29 14:34 containerd-shim-runc-v1
-rwxr-xr-x 1 root root  17M Jul 29 14:34 containerd-shim-runc-v2
root@containerd:~/workspace/containerd#

注意：编译过程中可能会因为缺少 btrfs 文件系统而报错（linux/btrfs_tree.h: No such file or directory）。遵循本文第一步的安装指导可避免此问题。⚠️

3 验证编译的二进制文件是否包含调试信息 🔍

验证编译是否成功并包含调试信息，可以通过以下任一方法：

3.1 使用objdump工具

成功编译的输出示例，会显示含调试信息的输出objdump --syms bin/containerd ✅

root@containerd:~/workspace/containerd# objdump --syms bin/containerd

bin/containerd:     file format elf64-x86-64

SYMBOL TABLE:
0000000000000000 l    df *ABS*  0000000000000000              Scrt1.o
00000000000003b4 l     O .note.ABI-tag  0000000000000020              __abi_tag
0000000000000000 l    df *ABS*  0000000000000000              go.go
00000000003ad4c0 l     F .text  0000000000000000              runtime.text
00000000003ad4c0 l     F .text  0000000000000059              internal/cpu.Initialize
00000000003ad520 l     F .text  0000000000000537              internal/cpu.processOptions
00000000003ada60 l     F .text  0000000000000026              internal/cpu.indexByte
00000000003adaa0 l     F .text  0000000000000925              internal/cpu.doinit
00000000003ae3e0 l     F .text  0000000000000006              internal/cpu.isSet
00000000003ae400 l     F .text  000000000000001b              internal/cpu.cpuid.abi0
00000000003ae420 l     F .text  0000000000000011              internal/cpu.xgetbv.abi0
00000000003ae440 l     F .text  0000000000000009              internal/cpu.getGOAMD64level.abi0
00000000003ae460 l     F .text  000000000000007a              type:.eq.internal/cpu.option
00000000003ae4e0 l     F .text  00000000000000e6              type:.eq.[6]internal/cpu.option
00000000003ae5e0 l     F .text  0000000000000003              runtime/internal/atomic.(*Int32).Load
00000000003ae600 l     F .text  0000000000000003              runtime/internal/atomic.(*Int32).Store
00000000003ae620 l     F .text  000000000000000d              runtime/internal/atomic.(*Int32).CompareAndSwap
00000000003ae640 l     F .text  000000000000000a              runtime/internal/atomic.(*Int32).Add
00000000003ae660 l     F .text  0000000000000004              runtime/internal/atomic.(*Int64).Load
00000000003ae680 l     F .text  0000000000000004              runtime/internal/atomic.(*Int64).Store
00000000003ae6a0 l     F .text  000000000000000f              runtime/internal/atomic.(*Int64).CompareAndSwap
00000000003ae6c0 l     F .text  0000000000000007              runtime/internal/atomic.(*Int64).Swap
00000000003ae6e0 l     F .text  000000000000000d              runtime/internal/atomic.(*Int64).Add
00000000003ae700 l     F .text  0000000000000003              runtime/internal/atomic.(*Uint8).Load
00000000003ae720 l     F .text  0000000000000003              runtime/internal/atomic.(*Uint8).Store
00000000003ae740 l     F .text  0000000000000004              runtime/internal/atomic.(*Uint8).And
00000000003ae760 l     F .text  0000000000000004              runtime/internal/atomic.(*Uint8).Or
00000000003ae780 l     F .text  0000000000000024              runtime/internal/atomic.(*Bool).Load
00000000003ae7c0 l     F .text  000000000000001f              runtime/internal/atomic.(*Bool).Store
00000000003ae7e0 l     F .text  0000000000000003              runtime/internal/atomic.(*Uint32).Load
00000000003ae800 l     F .text  0000000000000003              runtime/internal/atomic.(*Uint32).LoadAcquire
00000000003ae820 l     F .text  0000000000000003              runtime/internal/atomic.(*Uint32).Store
00000000003ae840 l     F .text  0000000000000003              runtime/internal/atomic.(*Uint32).StoreRelease
00000000003ae860 l     F .text  000000000000000d              runtime/internal/atomic.(*Uint32).CompareAndSwap
00000000003ae880 l     F .text  000000000000000d              runtime/internal/atomic.(*Uint32).CompareAndSwapRelease
00000000003ae8a0 l     F .text  0000000000000005              runtime/internal/atomic.(*Uint32).Swap
00000000003ae8c0 l     F .text  0000000000000004              runtime/internal/atomic.(*Uint32).And
00000000003ae8e0 l     F .text  0000000000000004              runtime/internal/atomic.(*Uint32).Or

若输出提示 no symbols，则表示该二进制文件不包含调试信息。❌

root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# objdump --syms /usr/local/bin/containerd

/usr/local/bin/containerd:     file format elf64-x86-64

SYMBOL TABLE:
no symbols


root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#

3.2 使用file工具

含调试信息的文件将显示 with debug_info, not stripped。✅

root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# file bin/containerd
bin/containerd: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=9bb291e61e1eceb23359dc29100845e5c1edf763, for GNU/Linux 3.2.0, with debug_info, not stripped
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#

若显示 stripped，则不包含调试信息。❌

root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# file /usr/local/bin/containerd
/usr/local/bin/containerd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=322f89b7e351fe2ccfaa0fe30de79c76d49d6e26, for GNU/Linux 3.2.0, stripped
root@containerd:~/workspace/containerd#

3.3 dlv工具验证

正确的调试信息如下所示：✅

root@containerd:~/workspace/containerd# dlv exec  bin/containerd
Type 'help' for list of commands.
(dlv)
(dlv)
(dlv)
(dlv)
(dlv)
(dlv) exit
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#

错误的调试信息如下，会提示：no debug info found，如下所示：❌

root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# dlv exec  /usr/local/bin/containerd
Warning: no debug info found, some functionality will be missing such as stack traces and variable evaluation.
Type 'help' for list of commands.
(dlv)
(dlv)
(dlv)
(dlv)
(dlv) exit

4 debug

debug containerd步骤如下：

• 1、利用ps -ef|gerp containerd查看containerd启动所需要的参数，这一步特别重要，尤其是在debug k8s源码的时候，k8s的每一个组件都带了很多命令行参数，想要调试这些组件，必须把这些组件的启动参数原封不动的加入到dlv调试命令当中
  • 注意，实际上执行上述命令之后会发现，containerd并没有启动参数，因此无需关心。
• 2、通过dlv命令启动containerd，启动命令我们可以从IDEA remote debug功能拷贝过来🔍
  • dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd📡
  • 注意：这里在启动contaienrd的时候没有指定任何参数，实际上也可以根据自己的情况加入containerd参数，譬如指定containerd的配置文件的位置（containerd默认配置文件为：/etc/containerd/config.toml），也可以指定调试的debug级别，譬如：
    • dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --config=/root/mycontainerd/config.toml --log-level=debug
    • dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --log-level=debug
• 3、在IDEA启动debug，连接到远程调试📍
  • 注意：在启动IDEA调试之前，你需要在想要debug的位置增加断点，否则程序启动会直接运行起来，等你这个时候打断点，很可能就晚了。
  • 注意：在启动IDEA调试之前，你需要修改IDEA操作系统标识为Linux

执行dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --log-level=debug命令之后，此时会阻塞在这里，千万不要使用ctrl + c，只有当IDEA连接上来的时候才会开始执行🚫⌛

root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --log-level=debug
API server listening at: [::]:12345
2023-07-29T15:06:45+08:00 warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)