追踪源码揪出mysql崩溃的真凶

本文涉及的产品
RDS MySQL Serverless 基础系列,0.5-2RCU 50GB
云数据库 RDS MySQL,集群系列 2核4GB
推荐场景:
搭建个人博客
云数据库 RDS MySQL,高可用系列 2核4GB
简介: 前言       在MySQL服务器断电或者kill mysqld进程后,再操作数据库会发生莫名其妙的重启,我们下面就根据一个案例,来逐步的进行分析和诊断,解决这种常见的问题。

前言

      在MySQL服务器断电或者kill mysqld进程后,再操作数据库会发生莫名其妙的重启,我们下面就根据一个案例,来逐步的进行分析和诊断,解决这种常见的问题。

故障现象

从一张表中查询数据时报如下错误



随后整理表时也报同样错误:

mysql> optimize table tablex ;

ERROR 2013 (HY000): Lost connection to MySQL server during query

这意味着无法直接对表进行修复工作。

如果加上limit从表中直接查询数据(select * from tablex limit 10000,1;),limit的值增长至198900时,重现上述错误。


故障初步分析

1.我们首先来看下当时的错误日志:

点击(此处)折叠或打开

  1. The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
  2. information that should help you find out what is causing the crash.
  3. 151203 12:31:23 mysqld_safe Number of processes running now: 0
  4. 151203 12:31:23 mysqld_safe mysqld restarted
  5. 2015-12-03 12:31:24 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
  6. 2015-12-03 12:31:24 0 [Note] /usr/local/mysql/bin/mysqld (mysqld 5.6.26-log) starting as process 2207 ...
  7. 2015-12-03 12:31:24 2207 [Note] Plugin 'FEDERATED' is disabled.
  8. 2015-12-03 12:31:24 7ff8243a6720 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
  9. 2015-12-03 12:31:24 2207 [Note] InnoDB: Using atomics to ref count buffer pool pages
  10. 2015-12-03 12:31:24 2207 [Note] InnoDB: The InnoDB memory heap is disabled
  11. 2015-12-03 12:31:24 2207 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
  12. 2015-12-03 12:31:24 2207 [Note] InnoDB: Memory barrier is not used
  13. 2015-12-03 12:31:24 2207 [Note] InnoDB: Compressed tables use zlib 1.2.3
  14. 2015-12-03 12:31:24 2207 [Note] InnoDB: Using Linux native AIO
  15. 2015-12-03 12:31:24 2207 [Note] InnoDB: Using CPU crc32 instructions
  16. 2015-12-03 12:31:24 2207 [Note] InnoDB: Initializing buffer pool, size = 2.5G
  17. 2015-12-03 12:31:24 2207 [Note] InnoDB: Completed initialization of buffer pool
  18. 2015-12-03 12:31:24 2207 [Note] InnoDB: Highest supported file format is Barracuda.
  19. 2015-12-03 12:31:24 2207 [Note] InnoDB: The log sequence numbers 20043317101 and 20043317101 in ibdata files do not match the log sequence number 20540489747 in the ib_logfiles!
  20. 2015-12-03 12:31:24 2207 [Note] InnoDB: Database was not shutdown normally!
  21. 2015-12-03 12:31:24 2207 [Note] InnoDB: Starting crash recovery.
  22. 2015-12-03 12:31:24 2207 [Note] InnoDB: Reading tablespace information from the .ibd files...
  23. 2015-12-03 12:31:24 2207 [Note] InnoDB: Restoring possible half-written data pages
  24. 2015-12-03 12:31:24 2207 [Note] InnoDB: from the doublewrite buffer...
  25. InnoDB: Last MySQL binlog file position 0 47771, file name mysql-bin.000036
  26. 2015-12-03 12:31:24 2207 [Note] InnoDB: 128 rollback segment(s) are active.
  27. 2015-12-03 12:31:24 2207 [Note] InnoDB: Waiting for purge to start
  28. 2015-12-03 12:31:24 2207 [Note] InnoDB: 5.6.26 started; log sequence number 20540489747
  29. 2015-12-03 12:31:24 2207 [Note] Recovering after a crash using /data/mysql3306/logs/mysql-bin
  30. 2015-12-03 12:31:24 2207 [Note] Starting crash recovery...
  31. 2015-12-03 12:31:24 2207 [Note] Crash recovery finished.
  32. 2015-12-03 12:31:25 2207 [Note] Server hostname (bind-address): '*'; port: 3306
  33. 2015-12-03 12:31:25 2207 [Note] IPv6 is available.
  34. 2015-12-03 12:31:25 2207 [Note] - '::' resolves to '::';
  35. 2015-12-03 12:31:25 2207 [Note] Server socket created on IP: '::'.
  36. 2015-12-03 12:31:25 2207 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
  37. 2015-12-03 12:31:25 2207 [Note] Event Scheduler: Loaded 0 events
  38. 2015-12-03 12:31:25 2207 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
  39. Version: '5.6.26-log' socket: '/data/mysql3306/mysql.sock' port: 3306 MySQL Community Server (GPL)
  40. 2015-12-03 12:31:25 2207 [Note] Slave I/O thread: connected to master 'replication@192.168.30.218:3306',replication started in log 'mysql-bin.000020' at position 47355
  41. 2015-12-03 12:31:25 2207 [Warning] Slave SQL: If a crash happens this configuration does not guarantee that the relay log info will be consistent, Error_code: 0
  42. 2015-12-03 12:31:25 2207 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.000020' at position 47355, relay log './mysqld-relay.000103' position: 283
  43. 2015-12-03 14:13:15 7ff723baa700 InnoDB: Assertion failure in thread 140699433084672 in file btr0pcur.cc line 430
  44. InnoDB: Failing assertion: page_is_comp(next_page) == page_is_comp(page)
  45. InnoDB: We intentionally generate a memory trap.
  46. InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
  47. InnoDB: If you get repeated assertion failures or crashes, even
  48. InnoDB: immediately after the mysqld startup, there may be
  49. InnoDB: corruption in the InnoDB tablespace. Please refer to
  50. InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
  51. InnoDB: about forcing recovery.
  52. 06:13:15 UTC - mysqld got signal 6 ;
  53. This could be because you hit a bug. It is also possible that this binary
  54. or one of the libraries it was linked against is corrupt, improperly built,
  55. or misconfigured. This error can also be caused by malfunctioning hardware.
  56. We will try our best to scrape up some info that will hopefully help
  57. diagnose the problem, but since we have already crashed,
  58. something is definitely wrong and this may fail.
  59. key_buffer_size=8388608
  60. read_buffer_size=131072
  61. max_used_connections=51
  62. max_threads=5000
  63. thread_count=52
  64. connection_count=50
  65. It is possible that mysqld could use up to
  66. key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1992176 K bytes of memory
  67. Hope that's ok; if not, decrease some variables in the equation.
  68. Thread pointer: 0x65d2370
  69. Attempting backtrace. You can use the following information to find out
  70. where mysqld died. If you see no messages after this, something went
  71. terribly wrong...
  72. stack_bottom = 7ff723ba9e10 thread_stack 0x40000
  73. /usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x90d8e5]
  74. /usr/local/mysql/bin/mysqld(handle_fatal_signal+0x3d8)[0x674078]
  75. /lib64/libpthread.so.0(+0xf790)[0x7ff823f89790]
  76. /lib64/libc.so.6(gsignal+0x35)[0x7ff822c34625]
  77. /lib64/libc.so.6(abort+0x175)[0x7ff822c35e05]
  78. /usr/local/mysql/bin/mysqld[0xa29954]
  79. /usr/local/mysql/bin/mysqld[0x9cfd85]
  80. /usr/local/mysql/bin/mysqld[0x931341]
  81. /usr/local/mysql/bin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x64)[0x58c2d4]
  82. /usr/local/mysql/bin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x37)[0x83ebc7]
  83. /usr/local/mysql/bin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x181)[0x6d2691]
  84. /usr/local/mysql/bin/mysqld(_ZN4JOIN4execEv+0x391)[0x6d04b1]
  85. /usr/local/mysql/bin/mysqld[0x717329]
  86. /usr/local/mysql/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xbc)[0x717dec]
  87. /usr/local/mysql/bin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x175)[0x717ff5]
  88. /usr/local/mysql/bin/mysqld[0x6f26e9]
  89. /usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THD+0x3574)[0x6f6f14]
  90. /usr/local/mysql/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x338)[0x6faa48]
  91. /usr/local/mysql/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xc68)[0x6fbe28]
  92. /usr/local/mysql/bin/mysqld(_Z10do_commandP3THD+0xd7)[0x6fdc37]
  93. /usr/local/mysql/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x116)[0x6c4bb6]
  94. /usr/local/mysql/bin/mysqld(handle_one_connection+0x45)[0x6c4c95]
  95. /usr/local/mysql/bin/mysqld(pfs_spawn_thread+0x126)[0xaf92f6]
  96. /lib64/libpthread.so.0(+0x7a51)[0x7ff823f81a51]
  97. /lib64/libc.so.6(clone+0x6d)[0x7ff822cea93d]
  98. Trying to get some variables.
  99. Some pointers may be invalid and cause the dump to abort.
  100. Query (7ff6d8004df0): is an invalid pointer
  101. Connection ID (thread ID): 7
  102. Status: NOT_KILLED
  103. The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
  104. information that should help you find out what is causing the crash.
  105. 151203 14:13:15 mysqld_safe Number of processes running now: 0
  106. 151203 14:13:15 mysqld_safe mysqld restarted
  107. 2015-12-03 14:13:16 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
  108. 2015-12-03 14:13:16 0 [Note] /usr/local/mysql/bin/mysqld (mysqld 5.6.26-log) starting as process 2481 ...
  109. 2015-12-03 14:13:16 2481 [Note] Plugin 'FEDERATED' is disabled.
  110. 2015-12-03 14:13:16 7fec16233720 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
  111. 2015-12-03 14:13:16 2481 [Note] InnoDB: Using atomics to ref count buffer pool pages
  112. 2015-12-03 14:13:16 2481 [Note] InnoDB: The InnoDB memory heap is disabled
  113. 2015-12-03 14:13:16 2481 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
  114. 2015-12-03 14:13:16 2481 [Note] InnoDB: Memory barrier is not used
  115. 2015-12-03 14:13:16 2481 [Note] InnoDB: Compressed tables use zlib 1.2.3
  116. 2015-12-03 14:13:16 2481 [Note] InnoDB: Using Linux native AIO
  117. 2015-12-03 14:13:16 2481 [Note] InnoDB: Using CPU crc32 instructions
  118. 2015-12-03 14:13:16 2481 [Note] InnoDB: Initializing buffer pool, size = 2.5G
  119. 2015-12-03 14:13:17 2481 [Note] InnoDB: Completed initialization of buffer pool
  120. 2015-12-03 14:13:17 2481 [Note] InnoDB: Highest supported file format is Barracuda.
  121. 2015-12-03 14:13:17 2481 [Note] InnoDB: The log sequence numbers 20043317101 and 20043317101 in ibdata files do not match the log sequence number 20612097468 in the ib_logfiles!
  122. 2015-12-03 14:13:17 2481 [Note] InnoDB: Database was not shutdown normally!
  123. 2015-12-03 14:13:17 2481 [Note] InnoDB: Starting crash recovery.
  124. 2015-12-03 14:13:17 2481 [Note] InnoDB: Reading tablespace information from the .ibd files...
  125. 2015-12-03 14:13:17 2481 [Note] InnoDB: Restoring possible half-written data pages
  126. 2015-12-03 14:13:17 2481 [Note] InnoDB: from the doublewrite buffer...
  127. InnoDB: Last MySQL binlog file position 0 366526, file name mysql-bin.000037
  128. 2015-12-03 14:13:17 2481 [Note] InnoDB: 128 rollback segment(s) are active.
  129. 2015-12-03 14:13:17 2481 [Note] InnoDB: Waiting for purge to start
  130. 2015-12-03 14:13:17 2481 [Note] InnoDB: 5.6.26 started; log sequence number 20612097468
  131. 2015-12-03 14:13:17 2481 [Note] Recovering after a crash using /data/mysql3306/logs/mysql-bin
  132. 2015-12-03 14:13:17 2481 [Note] Starting crash recovery...
  133. 2015-12-03 14:13:17 2481 [Note] Crash recovery finished.
  134. 2015-12-03 14:13:17 2481 [Note] Server hostname (bind-address): '*'; port: 3306
  135. 2015-12-03 14:13:17 2481 [Note] IPv6 is available.
  136. 2015-12-03 14:13:17 2481 [Note] - '::' resolves to '::';
  137. 2015-12-03 14:13:17 2481 [Note] Server socket created on IP: '::'.
  138. 2015-12-03 14:13:17 2481 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
  139. 2015-12-03 14:13:17 2481 [Note] Event Scheduler: Loaded 0 events
  140. 2015-12-03 14:13:17 2481 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
  141. Version: '5.6.26-log' socket: '/data/mysql3306/mysql.sock' port: 3306 MySQL Community Server (GPL)
  142. 2015-12-03 14:13:17 2481 [Note] Slave I/O thread: connected to master 'replication@192.168.30.218:3306',replication started in log 'mysql-bin.000020' at position 409891
  143. 2015-12-03 14:13:17 2481 [Warning] Slave SQL: If a crash happens this configuration does not guarantee that the relay log info will be consistent, Error_code: 0
  144. 2015-12-03 14:13:17 2481 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.000020' at position 409891, relay log './mysqld-relay.000105' position: 283

主要注意其中几行:

InnoDB: Assertion failure in thread 140699433084672 in file btr0pcur.cc line 430
InnoDB: Failing assertion: page_is_comp(next_page) == page_is_comp(page)
InnoDB: We intentionally generate a memory trap.

我们只需要简单的了解一些源码相关的知识,就可以判断此类的问题。

相关知识

MySQL源码目录结构

源码目录结构请参考彭立勋的PPT
http://ofslides.com/plinux-6880/presentation-36798

故障深入分析

大致了解了mysql源码的目录结构,我们再来看报错的那个文件:btr0pcur.cc。

注释显示这是个和索引树的游标相关的文件。


430行所处的函数是btr_pcur_move_to_next_page()。这个函数的功能是:  移动持续的游标到下一条记录。

这时我们就不难理解为什么服务器掉电后,读取数据会发生崩溃。就是因为游标读取索引时遇到了损坏的数据块。

故障排除

确定了问题所在之后,我们就可以开始修复了。

1.在my.cnf中[mysqld]下添加innodb_force_recovery=1,然后重启mysql服务。

2.尝试使用mysqldump备份所有数据。如果备份失败,逐个增加innodb_force_recovery的值(最大为6)后再尝试备份。

3.将备份的数据在别处还原,并对还原的数据进行校验。

4.原mysql实例建议重建数据目录。因为服务器掉电不仅会损坏ibd文件,共享表空间ibdata和日志也可能有损坏。

注:

也可以使用创建MyISAM表的方式导出损坏的表的数据。但mysqldump是首选。

CREATE TABLE new_table ... ENGINE=MYISAM;

INSERT (IGNORE) INTO new_table SELECT * FROM old_table;

故障总结

1.要定期对数据进行备份

2.缓存的电池也要做好保养

3.不要轻易对mysqld进程做kill操作

相关实践学习
如何在云端创建MySQL数据库
开始实验后,系统会自动创建一台自建MySQL的 源数据库 ECS 实例和一台 目标数据库 RDS。
全面了解阿里云能为你做什么
阿里云在全球各地部署高效节能的绿色数据中心,利用清洁计算为万物互联的新世界提供源源不断的能源动力,目前开服的区域包括中国(华北、华东、华南、香港)、新加坡、美国(美东、美西)、欧洲、中东、澳大利亚、日本。目前阿里云的产品涵盖弹性计算、数据库、存储与CDN、分析与搜索、云通信、网络、管理与监控、应用服务、互联网中间件、移动服务、视频服务等。通过本课程,来了解阿里云能够为你的业务带来哪些帮助     相关的阿里云产品:云服务器ECS 云服务器 ECS(Elastic Compute Service)是一种弹性可伸缩的计算服务,助您降低 IT 成本,提升运维效率,使您更专注于核心业务创新。产品详情: https://www.aliyun.com/product/ecs
相关文章
|
4月前
|
存储 安全 Java
基于Java+MySQL停车场车位管理系统详细设计和实现(源码+LW+调试文档+讲解等)
基于Java+MySQL停车场车位管理系统详细设计和实现(源码+LW+调试文档+讲解等)
|
2月前
|
存储 自然语言处理 关系型数据库
MySQL全文索引源码剖析之Insert语句执行过程
【8月更文挑战第17天】在MySQL中,处理含全文索引的`INSERT`语句涉及多步骤。首先进行语法解析确认语句结构无误;接着语义分析检查数据是否符合表结构及约束。随后存储引擎执行插入操作,若涉及全文索引则进行分词处理,并更新倒排索引结构。此外,事务管理确保了操作的完整性和一致性。通过示例创建含全文索引的表并插入数据,可见MySQL如何高效地处理此类操作,有助于优化数据库性能和提升全文搜索效果。
|
2月前
|
NoSQL 关系型数据库 MySQL
SpringBoot 集成 SpringSecurity + MySQL + JWT 附源码,废话不多直接盘
SpringBoot 集成 SpringSecurity + MySQL + JWT 附源码,废话不多直接盘
93 2
|
2月前
|
关系型数据库 MySQL Linux
【一键解锁神秘力量!】CentOS 7 通过编译源码方式安装 MySQL 数据库 —— 从零到英雄的数据库安装实战秘籍!
【8月更文挑战第9天】随着业务增长,对数据库的需求日益提高。在 CentOS 7 中,通过编译源码安装 MySQL 可提供更高定制性和灵活性。本文详细介绍从准备环境、下载源码、配置编译参数到安装 MySQL 的全过程,并对比 RPM 包安装方法,帮助读者根据需求选择合适方案。实践时需注意备份数据、选择合适版本、确保安全性和调优性能等要点。
151 1
|
3月前
|
数据库
基于PHP+MYSQL开发制作的趣味测试网站源码
基于PHP+MYSQL开发制作的趣味测试网站源码。可在后台提前设置好缘分, 自己手动在数据库里修改数据,数据库里有就会优先查询数据库的信息, 没设置的话第一次查询缘分都是非常好的 95-99,第二次查就比较差 , 所以如果要你女朋友查询你的名字觉得很好 那就得是她第一反应是查和你的缘分, 如果查的是别人,那不好意思,第二个可能是你。
53 3
|
2月前
|
关系型数据库 MySQL Java
“惊呆了!无需改动Nacos源码,轻松实现SGJDBC连接MySQL?这操作太秀了,速来围观,错过等哭!”
【8月更文挑战第7天】在使用Nacos进行服务治理时,常需连接MySQL存储数据。使用特定的SGJDBC驱动连接MySQL时,一般无需修改Nacos源码。需确保SGJDBC已添加至类路径,并在Nacos配置文件中指定使用SGJDBC的JDBC URL。示例中展示如何配置Nacos使用MySQL及SGJDBC,并在应用中通过Nacos API获取配置信息建立数据库连接,实现灵活集成不同JDBC驱动的目标。
45 0
|
4月前
vue.js+node.js+mysql在线聊天室源码
vue.js+node.js+mysql在线聊天室源码 技术栈:vue.js+Element UI+node.js+socket.io+mysql
159 3
|
4月前
|
移动开发 小程序 关系型数据库
java+ IDEA+ Uniapp+ mysql医院3D智能导诊系统源码
这是一个基于Java、IDEA、Uniapp和MySQL的医院3D智能导诊系统,采用Springboot后端框架和Redis、Mybatis Plus、RocketMQ等技术。系统通过对话式交互,精准推荐就诊科室,解决患者挂号困扰。它还具备智能预问诊功能,提升诊疗效率和准确性,确保医生能快速了解患者详情。此系统还支持小程序和H5,方便患者使用。
47 0
|
5月前
|
监控 NoSQL Java
java云MES 系统源码Java+ springboot+ mysql 一款基于云计算技术的企业级生产管理系统
MES系统是生产企业对制造执行系统实施的重点在智能制造执行管理领域,而MES系统特点中的可伸缩、信息精确、开放、承接、安全等也传递出:MES在此管理领域中无可替代的“王者之尊”。MES制造执行系统特点集可伸缩性、精确性、开放性、承接性、经济性与安全性于一体,帮助企业解决生产中遇到的实际问题,降低运营成本,快速适应企业不断的制造执行管理需求,使得企业已有基础设施与一切可用资源实现高度集成,提升企业投资的有效性。
96 5
|
4月前
|
关系型数据库 MySQL Linux
centos linux mysql 5.5脚本全自动源码包 编译安装
centos linux mysql 5.5脚本全自动源码包 编译安装
46 0
下一篇
无影云桌面