基本参考twitter的实现,代码很简单,直接调用的breakpad的接口,关键是修改cmake文件比较蛋疼。
twitter的实现:https://github.com/twitter/mysql/commit/f95c5a49b4703779d05c200a9b282284248b7cb9
需要作部分修改,这里记录下我的操作步骤
1.下载google-breakpad,如果是在r1001之前的版本直接编译安装即可(未尝试),如果是checkout的最新版本,需要把源代码内的my_strchr全部替换掉,俺直接替换成bp_my_strchr,这个会和mysql的定义冲突。
2.配置环境变量BREAKPAD_ROOT,不然breakpad是不会编译到MySQL中的:
$echo $BREAKPAD_ROOT
/u01/project/breakpad
3.configure && make && make install
4. backport twitter的patch
有点小不一样,在MySQL5.5.20之前的版本,调用函数my_write_minidump是在mysqld.cc的handle_segfault中,而在之后的版本中,为了解决bug#54082,这部分代码独立在单独文件sql/signal_handler.cc文件中,因此需要在函数handle_fatal_signal中调用my_write_minidump
cmake加上 -DENABLED_EMBEDDED_SERVER:BOOL=ON
5.用前几天report的一个bug尝试了一把,可以看到所有线程的堆栈/寄存器信息都被保存了下来,文件大小才629k。
通过minidump_stackwalk解析出来的部分内容如下:
Thread 34 (crashed)
0 mysqld!google_breakpad::ExceptionHandler::WriteMinidump [exception_handler.cc : 534 + 0xd]
rbx = 0x0000000000000000 r12 = 0x0000000000000000
r13 = 0x0000000000000000 r14 = 0x0000000000000080
r15 = 0x00007f0834008f10 rip = 0x0000000000991d3c
rsp = 0x00007f0891da4240 rbp = 0x00007f0891da4910
Found by: given as instruction pointer in context
1 mysqld!google_breakpad::ExceptionHandler::WriteMinidump [exception_handler.cc : 513 + 0x7]
rbx = 0x00007f0891da48b0 r12 = 0x0000000000000000
r13 = 0x0000000000000000 r14 = 0x0000000000000080
r15 = 0x00007f0834008f10 rip = 0x00000000009928ad
rsp = 0x00007f0891da48a0 rbp = 0x00007f0891da4910
Found by: call frame info
2 mysqld!my_write_minidump [minidump.cc : 71 + 0x21]
rbx = 0x0000000002af23d0 r12 = 0x00007f0891da497f
r13 = 0x00007f0891da4970 r14 = 0x0000000000000080
r15 = 0x00007f0834008f10 rip = 0x00000000006737dc
rsp = 0x00007f0891da4970 rbp = 0x00007f0891da49a0
Found by: call frame info
3 mysqld!handle_segfault [mysqld.cc : 2566 + 0xb]
rbx = 0x000000000000000b r12 = 0x000000000c57b6b0
r13 = 0x00000000009a167e r14 = 0x0000000000000080
r15 = 0x00007f0834008f10 rip = 0x000000000054e38a
rsp = 0x00007f0891da49b0 rbp = 0x00007f0891da4a30
Found by: call frame info
4 libpthread-2.12.so + 0xf51f
rbx = 0x0000000000008810 r12 = 0x0000000000000000
r13 = 0x0000000000000000 r14 = 0x00000000000000b0
r15 = 0x00007f0834008f10 rip = 0x0000003888c0f520
rsp = 0x00007f0891da4a40 rbp = 0x00007f0891da4f20
Found by: call frame info
5 mysqld!mi_state_info_read [mi_open.c : 958 + 0x2b]
rip = 0x00000000008d1954 rsp = 0x00007f0891da4af0
rbp = 0x00007f0891da4f20
Found by: stack scanning
6 mysqld!mi_open [mi_open.c : 210 + 0xf]
rbx = 0x0000000000000000 r12 = 0x00007f0891dae840
r13 = 0x00007f0891dadeb0 r14 = 0x00000000000000b0
rip = 0x00000000008d3832 rsp = 0x00007f0891da4f30
rbp = 0x00007f0891daeb50
Found by: call frame info
7 libc-2.12.so + 0x79adc
rip = 0x0000003888879add rsp = 0x00007f0891da4f70
rbp = 0x00007f0891daeb50
Found by: stack scanning
8 mysqld!my_malloc [my_malloc.c : 38 + 0x7]
rip = 0x000000000077c6e2 rsp = 0x00007f0891da4f90
rbp = 0x00007f0891daeb50
Found by: stack scanning
9 mysqld!my_strdup [my_malloc.c : 147 + 0xd]
rip = 0x000000000077c822 rsp = 0x00007f0891da4fc0
rbp = 0x00007f0891daeb50
Found by: stack scanning
10 mysqld!my_register_filename [mysql_thread.h : 671 + 0x7]
rip = 0x000000000077cfb0 rsp = 0x00007f0891da4ff0
rbp = 0x00007f0891daeb50
Found by: stack scanning
Thread 0
0 libc-2.12.so + 0xdd1e3
rbx = 0x0000000000000002 r12 = 0x000000000000003b
r13 = 0x0000000000000012 r14 = 0x00007fffb1eee56c
r15 = 0x000000000c5530c0 rip = 0x00000038888dd1e3
rsp = 0x00007fffb1eee400 rbp = 0x00007fffb1eee5a0
Found by: given as instruction pointer in context
1 mysqld!handle_connections_sockets [mysqld.cc : 5376 + 0x14]
rip = 0x000000000054eaa4 rsp = 0x00007fffb1eee430
rbp = 0x00007fffb1eee5a0
Found by: stack scanning
2 mysqld!mysqld_main [mysqld.cc : 4815 + 0x4]
rbx = 0x00007fffb1eee5c0 r12 = 0x0000000000000000
r13 = 0x00007fffb1eee6e0 r14 = 0x0000000000000000
r15 = 0x0000000000000000 rip = 0x0000000000551ee6
rsp = 0x00007fffb1eee5b0 rbp = 0x00007fffb1eee600
Found by: call frame info
3 libc-2.12.so + 0x1ec9c
rbx = 0x0000000000000000 r12 = 0x0000000000549340
r13 = 0x00007fffb1eee6e0 r14 = 0x0000000000000000
r15 = 0x0000000000000000 rip = 0x000000388881ec9d
rsp = 0x00007fffb1eee610 rbp = 0x0000000000000000
Found by: call frame info
4 mysqld + 0x14942f
rip = 0x0000000000549430 rsp = 0x00007fffb1eee630
Found by: stack scanning
……
……
……
Thread 37
0 libpthread-2.12.so + 0xb44c
rbx = 0x0000000001049b00 r12 = 0x0000000001048de0
r13 = 0x00000000009a4588 r14 = 0x0000000000000879
r15 = 0x0000000000000003 rip = 0x0000003888c0b44c
rsp = 0x00007f08902b5d48 rbp = 0x00007f08902b5df0
Found by: given as instruction pointer in context
1 mysqld!inline_mysql_cond_wait [mysql_thread.h : 980 + 0xa]
rip = 0x000000000054d093 rsp = 0x00007f08902b5d70
rbp = 0x00007f08902b5df0
Found by: stack scanning
2 mysqld!cache_thread [mysqld.cc : 2169 + 0x18]
rbx = 0x0000000000000001 r12 = 0x00007f08902b5e58
r13 = 0x00007f08902b69c0 r14 = 0x0000000000000000
r15 = 0x0000000000000003 rip = 0x000000000054e5d9
rsp = 0x00007f08902b5e00 rbp = 0x00007f08902b5e20
Found by: call frame info
3 mysqld!one_thread_per_connection_end [mysqld.cc : 2234 + 0x4]
rbx = 0x0000000000000001 r12 = 0x00007f08902b5e58
r13 = 0x00007f08902b69c0 r14 = 0x0000000000000000
r15 = 0x0000000000000003 rip = 0x000000000054e79d
rsp = 0x00007f08902b5e30 rbp = 0x00007f08902b5e40
Found by: call frame info
4 mysqld!do_handle_one_connection [sql_connect.cc : 1418 + 0x1f]
rbx = 0x0000000000000001 r12 = 0x00007f08902b5e58
r13 = 0x00007f08902b69c0 r14 = 0x0000000000000000
r15 = 0x0000000000000003 rip = 0x0000000000638b8c
rsp = 0x00007f08902b5e50 rbp = 0x00007f08902b5e70
Found by: call frame info
5 mysqld!handle_one_connection [sql_connect.cc : 1315 + 0x7]
rbx = 0x000000000c5b1ce0 r12 = 0x0000000001049420
r13 = 0x00007f08902b69c0 r14 = 0x0000000000000000
r15 = 0x0000000000000003 rip = 0x0000000000638c30
rsp = 0x00007f08902b5e80 rbp = 0x00007f08902b5e90
Found by: call frame info
6 libpthread-2.12.so + 0x77e0
rbx = 0x0000000000000000 r12 = 0x0000000001049420
r13 = 0x00007f08902b69c0 r14 = 0x0000000000000000
r15 = 0x0000000000000003 rip = 0x0000003888c077e1
rsp = 0x00007f08902b5ea0 rbp = 0x0000000000000000
Found by: call frame info
Loaded modules:
0x00400000 – 0x00d99fff mysqld ??? (main)
0x3888000000 – 0x388801ffff ld-2.12.so ???
0x3888400000 – 0x3888603fff libdl-2.12.so ???
0x3888800000 – 0x3888b8bfff libc-2.12.so ???
0x3888c00000 – 0x3888e18fff libpthread-2.12.so ???
0x3889400000 – 0x3889683fff libm-2.12.so ???
0x3889800000 – 0x3889a07fff librt-2.12.so ???
0x388b400000 – 0x388b615fff libgcc_s-4.4.5-20110214.so.1 ???
0x388bc00000 – 0x388be5dfff libfreebl3.so ???
0x388c400000 – 0x388c608fff libcrypt-2.12.so ???
0x388e400000 – 0x388e6f0fff libstdc++.so.6.0.13 ???
0x7fffb1fff000 – 0x7fffb1ffffff linux-gate.so ???
可以看到,保留的信息还是相当丰富,足以用于诊断问题,并且文件很小(629k),相应的core file则有6G。
———————————-