追踪源码揪出mysql崩溃的真凶

本文涉及的产品
RDS MySQL Serverless 基础系列,0.5-2RCU 50GB
云数据库 RDS MySQL,集群系列 2核4GB
推荐场景:
搭建个人博客
云数据库 RDS MySQL,高可用系列 2核4GB
简介: 前言       在MySQL服务器断电或者kill mysqld进程后,再操作数据库会发生莫名其妙的重启,我们下面就根据一个案例,来逐步的进行分析和诊断,解决这种常见的问题。

前言

      在MySQL服务器断电或者kill mysqld进程后,再操作数据库会发生莫名其妙的重启,我们下面就根据一个案例,来逐步的进行分析和诊断,解决这种常见的问题。

故障现象

从一张表中查询数据时报如下错误



随后整理表时也报同样错误:

mysql> optimize table tablex ;

ERROR 2013 (HY000): Lost connection to MySQL server during query

这意味着无法直接对表进行修复工作。

如果加上limit从表中直接查询数据(select * from tablex limit 10000,1;),limit的值增长至198900时,重现上述错误。


故障初步分析

1.我们首先来看下当时的错误日志:

点击(此处)折叠或打开

  1. The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
  2. information that should help you find out what is causing the crash.
  3. 151203 12:31:23 mysqld_safe Number of processes running now: 0
  4. 151203 12:31:23 mysqld_safe mysqld restarted
  5. 2015-12-03 12:31:24 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
  6. 2015-12-03 12:31:24 0 [Note] /usr/local/mysql/bin/mysqld (mysqld 5.6.26-log) starting as process 2207 ...
  7. 2015-12-03 12:31:24 2207 [Note] Plugin 'FEDERATED' is disabled.
  8. 2015-12-03 12:31:24 7ff8243a6720 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
  9. 2015-12-03 12:31:24 2207 [Note] InnoDB: Using atomics to ref count buffer pool pages
  10. 2015-12-03 12:31:24 2207 [Note] InnoDB: The InnoDB memory heap is disabled
  11. 2015-12-03 12:31:24 2207 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
  12. 2015-12-03 12:31:24 2207 [Note] InnoDB: Memory barrier is not used
  13. 2015-12-03 12:31:24 2207 [Note] InnoDB: Compressed tables use zlib 1.2.3
  14. 2015-12-03 12:31:24 2207 [Note] InnoDB: Using Linux native AIO
  15. 2015-12-03 12:31:24 2207 [Note] InnoDB: Using CPU crc32 instructions
  16. 2015-12-03 12:31:24 2207 [Note] InnoDB: Initializing buffer pool, size = 2.5G
  17. 2015-12-03 12:31:24 2207 [Note] InnoDB: Completed initialization of buffer pool
  18. 2015-12-03 12:31:24 2207 [Note] InnoDB: Highest supported file format is Barracuda.
  19. 2015-12-03 12:31:24 2207 [Note] InnoDB: The log sequence numbers 20043317101 and 20043317101 in ibdata files do not match the log sequence number 20540489747 in the ib_logfiles!
  20. 2015-12-03 12:31:24 2207 [Note] InnoDB: Database was not shutdown normally!
  21. 2015-12-03 12:31:24 2207 [Note] InnoDB: Starting crash recovery.
  22. 2015-12-03 12:31:24 2207 [Note] InnoDB: Reading tablespace information from the .ibd files...
  23. 2015-12-03 12:31:24 2207 [Note] InnoDB: Restoring possible half-written data pages
  24. 2015-12-03 12:31:24 2207 [Note] InnoDB: from the doublewrite buffer...
  25. InnoDB: Last MySQL binlog file position 0 47771, file name mysql-bin.000036
  26. 2015-12-03 12:31:24 2207 [Note] InnoDB: 128 rollback segment(s) are active.
  27. 2015-12-03 12:31:24 2207 [Note] InnoDB: Waiting for purge to start
  28. 2015-12-03 12:31:24 2207 [Note] InnoDB: 5.6.26 started; log sequence number 20540489747
  29. 2015-12-03 12:31:24 2207 [Note] Recovering after a crash using /data/mysql3306/logs/mysql-bin
  30. 2015-12-03 12:31:24 2207 [Note] Starting crash recovery...
  31. 2015-12-03 12:31:24 2207 [Note] Crash recovery finished.
  32. 2015-12-03 12:31:25 2207 [Note] Server hostname (bind-address): '*'; port: 3306
  33. 2015-12-03 12:31:25 2207 [Note] IPv6 is available.
  34. 2015-12-03 12:31:25 2207 [Note] - '::' resolves to '::';
  35. 2015-12-03 12:31:25 2207 [Note] Server socket created on IP: '::'.
  36. 2015-12-03 12:31:25 2207 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
  37. 2015-12-03 12:31:25 2207 [Note] Event Scheduler: Loaded 0 events
  38. 2015-12-03 12:31:25 2207 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
  39. Version: '5.6.26-log' socket: '/data/mysql3306/mysql.sock' port: 3306 MySQL Community Server (GPL)
  40. 2015-12-03 12:31:25 2207 [Note] Slave I/O thread: connected to master 'replication@192.168.30.218:3306',replication started in log 'mysql-bin.000020' at position 47355
  41. 2015-12-03 12:31:25 2207 [Warning] Slave SQL: If a crash happens this configuration does not guarantee that the relay log info will be consistent, Error_code: 0
  42. 2015-12-03 12:31:25 2207 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.000020' at position 47355, relay log './mysqld-relay.000103' position: 283
  43. 2015-12-03 14:13:15 7ff723baa700 InnoDB: Assertion failure in thread 140699433084672 in file btr0pcur.cc line 430
  44. InnoDB: Failing assertion: page_is_comp(next_page) == page_is_comp(page)
  45. InnoDB: We intentionally generate a memory trap.
  46. InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
  47. InnoDB: If you get repeated assertion failures or crashes, even
  48. InnoDB: immediately after the mysqld startup, there may be
  49. InnoDB: corruption in the InnoDB tablespace. Please refer to
  50. InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
  51. InnoDB: about forcing recovery.
  52. 06:13:15 UTC - mysqld got signal 6 ;
  53. This could be because you hit a bug. It is also possible that this binary
  54. or one of the libraries it was linked against is corrupt, improperly built,
  55. or misconfigured. This error can also be caused by malfunctioning hardware.
  56. We will try our best to scrape up some info that will hopefully help
  57. diagnose the problem, but since we have already crashed,
  58. something is definitely wrong and this may fail.
  59. key_buffer_size=8388608
  60. read_buffer_size=131072
  61. max_used_connections=51
  62. max_threads=5000
  63. thread_count=52
  64. connection_count=50
  65. It is possible that mysqld could use up to
  66. key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1992176 K bytes of memory
  67. Hope that's ok; if not, decrease some variables in the equation.
  68. Thread pointer: 0x65d2370
  69. Attempting backtrace. You can use the following information to find out
  70. where mysqld died. If you see no messages after this, something went
  71. terribly wrong...
  72. stack_bottom = 7ff723ba9e10 thread_stack 0x40000
  73. /usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x90d8e5]
  74. /usr/local/mysql/bin/mysqld(handle_fatal_signal+0x3d8)[0x674078]
  75. /lib64/libpthread.so.0(+0xf790)[0x7ff823f89790]
  76. /lib64/libc.so.6(gsignal+0x35)[0x7ff822c34625]
  77. /lib64/libc.so.6(abort+0x175)[0x7ff822c35e05]
  78. /usr/local/mysql/bin/mysqld[0xa29954]
  79. /usr/local/mysql/bin/mysqld[0x9cfd85]
  80. /usr/local/mysql/bin/mysqld[0x931341]
  81. /usr/local/mysql/bin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x64)[0x58c2d4]
  82. /usr/local/mysql/bin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x37)[0x83ebc7]
  83. /usr/local/mysql/bin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x181)[0x6d2691]
  84. /usr/local/mysql/bin/mysqld(_ZN4JOIN4execEv+0x391)[0x6d04b1]
  85. /usr/local/mysql/bin/mysqld[0x717329]
  86. /usr/local/mysql/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xbc)[0x717dec]
  87. /usr/local/mysql/bin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x175)[0x717ff5]
  88. /usr/local/mysql/bin/mysqld[0x6f26e9]
  89. /usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THD+0x3574)[0x6f6f14]
  90. /usr/local/mysql/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x338)[0x6faa48]
  91. /usr/local/mysql/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xc68)[0x6fbe28]
  92. /usr/local/mysql/bin/mysqld(_Z10do_commandP3THD+0xd7)[0x6fdc37]
  93. /usr/local/mysql/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x116)[0x6c4bb6]
  94. /usr/local/mysql/bin/mysqld(handle_one_connection+0x45)[0x6c4c95]
  95. /usr/local/mysql/bin/mysqld(pfs_spawn_thread+0x126)[0xaf92f6]
  96. /lib64/libpthread.so.0(+0x7a51)[0x7ff823f81a51]
  97. /lib64/libc.so.6(clone+0x6d)[0x7ff822cea93d]
  98. Trying to get some variables.
  99. Some pointers may be invalid and cause the dump to abort.
  100. Query (7ff6d8004df0): is an invalid pointer
  101. Connection ID (thread ID): 7
  102. Status: NOT_KILLED
  103. The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
  104. information that should help you find out what is causing the crash.
  105. 151203 14:13:15 mysqld_safe Number of processes running now: 0
  106. 151203 14:13:15 mysqld_safe mysqld restarted
  107. 2015-12-03 14:13:16 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
  108. 2015-12-03 14:13:16 0 [Note] /usr/local/mysql/bin/mysqld (mysqld 5.6.26-log) starting as process 2481 ...
  109. 2015-12-03 14:13:16 2481 [Note] Plugin 'FEDERATED' is disabled.
  110. 2015-12-03 14:13:16 7fec16233720 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
  111. 2015-12-03 14:13:16 2481 [Note] InnoDB: Using atomics to ref count buffer pool pages
  112. 2015-12-03 14:13:16 2481 [Note] InnoDB: The InnoDB memory heap is disabled
  113. 2015-12-03 14:13:16 2481 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
  114. 2015-12-03 14:13:16 2481 [Note] InnoDB: Memory barrier is not used
  115. 2015-12-03 14:13:16 2481 [Note] InnoDB: Compressed tables use zlib 1.2.3
  116. 2015-12-03 14:13:16 2481 [Note] InnoDB: Using Linux native AIO
  117. 2015-12-03 14:13:16 2481 [Note] InnoDB: Using CPU crc32 instructions
  118. 2015-12-03 14:13:16 2481 [Note] InnoDB: Initializing buffer pool, size = 2.5G
  119. 2015-12-03 14:13:17 2481 [Note] InnoDB: Completed initialization of buffer pool
  120. 2015-12-03 14:13:17 2481 [Note] InnoDB: Highest supported file format is Barracuda.
  121. 2015-12-03 14:13:17 2481 [Note] InnoDB: The log sequence numbers 20043317101 and 20043317101 in ibdata files do not match the log sequence number 20612097468 in the ib_logfiles!
  122. 2015-12-03 14:13:17 2481 [Note] InnoDB: Database was not shutdown normally!
  123. 2015-12-03 14:13:17 2481 [Note] InnoDB: Starting crash recovery.
  124. 2015-12-03 14:13:17 2481 [Note] InnoDB: Reading tablespace information from the .ibd files...
  125. 2015-12-03 14:13:17 2481 [Note] InnoDB: Restoring possible half-written data pages
  126. 2015-12-03 14:13:17 2481 [Note] InnoDB: from the doublewrite buffer...
  127. InnoDB: Last MySQL binlog file position 0 366526, file name mysql-bin.000037
  128. 2015-12-03 14:13:17 2481 [Note] InnoDB: 128 rollback segment(s) are active.
  129. 2015-12-03 14:13:17 2481 [Note] InnoDB: Waiting for purge to start
  130. 2015-12-03 14:13:17 2481 [Note] InnoDB: 5.6.26 started; log sequence number 20612097468
  131. 2015-12-03 14:13:17 2481 [Note] Recovering after a crash using /data/mysql3306/logs/mysql-bin
  132. 2015-12-03 14:13:17 2481 [Note] Starting crash recovery...
  133. 2015-12-03 14:13:17 2481 [Note] Crash recovery finished.
  134. 2015-12-03 14:13:17 2481 [Note] Server hostname (bind-address): '*'; port: 3306
  135. 2015-12-03 14:13:17 2481 [Note] IPv6 is available.
  136. 2015-12-03 14:13:17 2481 [Note] - '::' resolves to '::';
  137. 2015-12-03 14:13:17 2481 [Note] Server socket created on IP: '::'.
  138. 2015-12-03 14:13:17 2481 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
  139. 2015-12-03 14:13:17 2481 [Note] Event Scheduler: Loaded 0 events
  140. 2015-12-03 14:13:17 2481 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
  141. Version: '5.6.26-log' socket: '/data/mysql3306/mysql.sock' port: 3306 MySQL Community Server (GPL)
  142. 2015-12-03 14:13:17 2481 [Note] Slave I/O thread: connected to master 'replication@192.168.30.218:3306',replication started in log 'mysql-bin.000020' at position 409891
  143. 2015-12-03 14:13:17 2481 [Warning] Slave SQL: If a crash happens this configuration does not guarantee that the relay log info will be consistent, Error_code: 0
  144. 2015-12-03 14:13:17 2481 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.000020' at position 409891, relay log './mysqld-relay.000105' position: 283

主要注意其中几行:

InnoDB: Assertion failure in thread 140699433084672 in file btr0pcur.cc line 430
InnoDB: Failing assertion: page_is_comp(next_page) == page_is_comp(page)
InnoDB: We intentionally generate a memory trap.

我们只需要简单的了解一些源码相关的知识,就可以判断此类的问题。

相关知识

MySQL源码目录结构

源码目录结构请参考彭立勋的PPT
http://ofslides.com/plinux-6880/presentation-36798

故障深入分析

大致了解了mysql源码的目录结构,我们再来看报错的那个文件:btr0pcur.cc。

注释显示这是个和索引树的游标相关的文件。


430行所处的函数是btr_pcur_move_to_next_page()。这个函数的功能是:  移动持续的游标到下一条记录。

这时我们就不难理解为什么服务器掉电后,读取数据会发生崩溃。就是因为游标读取索引时遇到了损坏的数据块。

故障排除

确定了问题所在之后,我们就可以开始修复了。

1.在my.cnf中[mysqld]下添加innodb_force_recovery=1,然后重启mysql服务。

2.尝试使用mysqldump备份所有数据。如果备份失败,逐个增加innodb_force_recovery的值(最大为6)后再尝试备份。

3.将备份的数据在别处还原,并对还原的数据进行校验。

4.原mysql实例建议重建数据目录。因为服务器掉电不仅会损坏ibd文件,共享表空间ibdata和日志也可能有损坏。

注:

也可以使用创建MyISAM表的方式导出损坏的表的数据。但mysqldump是首选。

CREATE TABLE new_table ... ENGINE=MYISAM;

INSERT (IGNORE) INTO new_table SELECT * FROM old_table;

故障总结

1.要定期对数据进行备份

2.缓存的电池也要做好保养

3.不要轻易对mysqld进程做kill操作

相关实践学习
如何快速连接云数据库RDS MySQL
本场景介绍如何通过阿里云数据管理服务DMS快速连接云数据库RDS MySQL,然后进行数据表的CRUD操作。
全面了解阿里云能为你做什么
阿里云在全球各地部署高效节能的绿色数据中心,利用清洁计算为万物互联的新世界提供源源不断的能源动力,目前开服的区域包括中国(华北、华东、华南、香港)、新加坡、美国(美东、美西)、欧洲、中东、澳大利亚、日本。目前阿里云的产品涵盖弹性计算、数据库、存储与CDN、分析与搜索、云通信、网络、管理与监控、应用服务、互联网中间件、移动服务、视频服务等。通过本课程,来了解阿里云能够为你的业务带来哪些帮助     相关的阿里云产品:云服务器ECS 云服务器 ECS(Elastic Compute Service)是一种弹性可伸缩的计算服务,助您降低 IT 成本,提升运维效率,使您更专注于核心业务创新。产品详情: https://www.aliyun.com/product/ecs
相关文章
|
19天前
|
JavaScript 安全 Java
java版药品不良反应智能监测系统源码,采用SpringBoot、Vue、MySQL技术开发
基于B/S架构,采用Java、SpringBoot、Vue、MySQL等技术自主研发的ADR智能监测系统,适用于三甲医院,支持二次开发。该系统能自动监测全院患者药物不良反应,通过移动端和PC端实时反馈,提升用药安全。系统涵盖规则管理、监测报告、系统管理三大模块,确保精准、高效地处理ADR事件。
|
1月前
|
关系型数据库 MySQL Linux
在 CentOS 7 中通过编译源码方式安装 MySQL 数据库的详细步骤,并与使用 RPM 包安装进行了对比
本文介绍了在 CentOS 7 中通过编译源码方式安装 MySQL 数据库的详细步骤,并与使用 RPM 包安装进行了对比。通过具体案例,读者可以了解如何准备环境、下载源码、编译安装、配置服务及登录 MySQL。编译源码安装虽然复杂,但提供了更高的定制性和灵活性,适用于需要高度定制的场景。
112 3
|
1月前
|
关系型数据库 MySQL Linux
在 CentOS 7 中通过编译源码方式安装 MySQL 数据库的详细步骤,包括准备工作、下载源码、编译安装、配置 MySQL 服务、登录设置等。
本文介绍了在 CentOS 7 中通过编译源码方式安装 MySQL 数据库的详细步骤,包括准备工作、下载源码、编译安装、配置 MySQL 服务、登录设置等。同时,文章还对比了编译源码安装与使用 RPM 包安装的优缺点,帮助读者根据需求选择最合适的方法。通过具体案例,展示了编译源码安装的灵活性和定制性。
158 2
|
2月前
|
关系型数据库 MySQL Linux
在 CentOS 7 中通过编译源码方式安装 MySQL 数据库的详细步骤
本文介绍了在 CentOS 7 中通过编译源码方式安装 MySQL 数据库的详细步骤,包括准备工作、下载源码、编译安装、配置服务等,并与使用 RPM 包安装进行了对比,帮助读者根据需求选择合适的方法。编译源码安装虽然复杂,但提供了更高的定制性和灵活性。
289 2
|
2月前
|
关系型数据库 MySQL Linux
在 CentOS 7 中通过编译源码方式安装 MySQL 数据库的详细步骤
【10月更文挑战第7天】本文介绍了在 CentOS 7 中通过编译源码方式安装 MySQL 数据库的详细步骤,包括准备工作、下载源码、编译安装、配置 MySQL 服务、登录设置等。同时,文章还对比了编译源码安装与使用 RPM 包安装的优缺点,帮助读者根据自身需求选择合适的方法。
65 3
|
2月前
|
前端开发 Java 数据库连接
表白墙/留言墙 —— 中级SpringBoot项目,MyBatis技术栈MySQL数据库开发,练手项目前后端开发(带完整源码) 全方位全步骤手把手教学
本文是一份全面的表白墙/留言墙项目教程,使用SpringBoot + MyBatis技术栈和MySQL数据库开发,涵盖了项目前后端开发、数据库配置、代码实现和运行的详细步骤。
77 0
表白墙/留言墙 —— 中级SpringBoot项目,MyBatis技术栈MySQL数据库开发,练手项目前后端开发(带完整源码) 全方位全步骤手把手教学
|
4月前
|
存储 自然语言处理 关系型数据库
MySQL全文索引源码剖析之Insert语句执行过程
【8月更文挑战第17天】在MySQL中,处理含全文索引的`INSERT`语句涉及多步骤。首先进行语法解析确认语句结构无误;接着语义分析检查数据是否符合表结构及约束。随后存储引擎执行插入操作,若涉及全文索引则进行分词处理,并更新倒排索引结构。此外,事务管理确保了操作的完整性和一致性。通过示例创建含全文索引的表并插入数据,可见MySQL如何高效地处理此类操作,有助于优化数据库性能和提升全文搜索效果。
|
4月前
|
NoSQL 关系型数据库 MySQL
SpringBoot 集成 SpringSecurity + MySQL + JWT 附源码,废话不多直接盘
SpringBoot 集成 SpringSecurity + MySQL + JWT 附源码,废话不多直接盘
173 2
|
4月前
|
关系型数据库 MySQL Linux
【一键解锁神秘力量!】CentOS 7 通过编译源码方式安装 MySQL 数据库 —— 从零到英雄的数据库安装实战秘籍!
【8月更文挑战第9天】随着业务增长,对数据库的需求日益提高。在 CentOS 7 中,通过编译源码安装 MySQL 可提供更高定制性和灵活性。本文详细介绍从准备环境、下载源码、配置编译参数到安装 MySQL 的全过程,并对比 RPM 包安装方法,帮助读者根据需求选择合适方案。实践时需注意备份数据、选择合适版本、确保安全性和调优性能等要点。
232 1
|
4月前
|
关系型数据库 MySQL Java
“惊呆了!无需改动Nacos源码,轻松实现SGJDBC连接MySQL?这操作太秀了,速来围观,错过等哭!”
【8月更文挑战第7天】在使用Nacos进行服务治理时,常需连接MySQL存储数据。使用特定的SGJDBC驱动连接MySQL时,一般无需修改Nacos源码。需确保SGJDBC已添加至类路径,并在Nacos配置文件中指定使用SGJDBC的JDBC URL。示例中展示如何配置Nacos使用MySQL及SGJDBC,并在应用中通过Nacos API获取配置信息建立数据库连接,实现灵活集成不同JDBC驱动的目标。
133 0

推荐镜像

更多