MySQL运维之 binlog_gtid_simple_recovery(GTID)

简介:

binlog_gtid_simple_recovery 是什么

  • 官方解释
This variable controls how binary log files are iterated during the search for GTIDs when MySQL starts or restarts.
In MySQL version 5.7.5, this variable was added as simplified_binlog_gtid_recovery and in MySQL version 5.7.6 it was renamed to binlog_gtid_simple_recovery.

When binlog_gtid_simple_recovery=FALSE, the method of iterating the binary log files is:

To initialize gtid_executed, binary log files are iterated from the newest file, stopping at the first binary log that has any Previous_gtids_log_event.
All GTIDs from Previous_gtids_log_event and Gtid_log_events are read from this binary log file.
This GTID set is stored internally and called gtids_in_binlog.
The value of gtid_executed is computed as the union of this set and the GTIDs stored in the mysql.gtid_executed table.

This process could take a long time if you had a large number of binary log files without GTID events, for example created when gtid_mode=OFF.

To initialize gtid_purged, binary log files are iterated from the oldest to the newest,
stopping at the first binary log that contains either a Previous_gtids_log_event that is nonempty (that has at least one GTID)
or that has at least one Gtid_log_event. From this binary log it reads Previous_gtids_log_event.
This GTID set is subtracted from gtids_in_binlog and the result stored in the internal variable gtids_in_binlog_not_purged.
The value of gtid_purged is initialized to the value of gtid_executed, minus gtids_in_binlog_not_purged.

When binlog_gtid_simple_recovery=TRUE, which is the default in MySQL 5.7.7 and later,
the server iterates only the oldest and the newest binary log files
and the values of gtid_purged and gtid_executed are computed based only on Previous_gtids_log_event or Gtid_log_event found in these files.
This ensures only two binary log files are iterated during server restart or when binary logs are being purged.
  • 官方注意点
Note

If this option is enabled, gtid_executed and gtid_purged may be initialized incorrectly in the following situations:

The newest binary log was generated by MySQL 5.7.5 or older, and gtid_mode was ON for some binary logs but OFF for the newest binary log.

A SET GTID_PURGED statement was issued on a MySQL version prior to 5.7.7, and the binary log that was active at the time of the SET GTID_PURGED has not yet been purged.

If an incorrect GTID set is computed in either situation, it will remain incorrect even if the server is later restarted, regardless of the value of this option.
  • 个人理解与总结
1. 这个变量用于在MySQL重启或启动的时候寻找GTIDs过程中,控制binlog 如何遍历的算法?
2. 当binlog_gtid_simple_recovery=FALSE 时:
    为了初始化 gtid_executed,算法是: 从newest_binlog -> oldest_binlog 方向遍历读取,如果发现有Previous_gtids_log_event , 那么就停止遍历
    为了初始化 gtid_purged,算法是:   从oldest_binlog -> newest_binlog 方向遍历读取, 如果发现有Previous_gtids_log_event(not empty)或者 至少有一个Gtid_log_event的文件,那么就停止遍历
3. 当binlog_gtid_simple_recovery=TRUE 时:
    为了初始化 gtid_executed , 算法是: 只需要读取newest_binlog
    为了初始化 gtid_purged, 算法是: 只需要读取oldest_binlog
4. 当设置binlog_gtid_simple_recovery=TRUE , 如果MySQL版本低于5.7.7 , 可能会有gitd计算出错的可能,具体参考官方文档详细描述

根据以上解读,那么如果存在非gtid的binlog比较多的时候,会非常影响性能的。
接下来,我们就来好好测试这种场景

测试案例

重点测试non-gtid和gtid混合的情况: This process could take a long time if you had a large number of binary log files without GTID events, for example created when gtid_mode=OFF.
测试当删除binlog的时候,是如何重置gtid_purged值的

  • 环境
    MySQL5.7.13
    binlog_gtid_simple_recovery = false   => 这是重点
    GTID升级:non-GTID -> GTID 后,purge binary logs  => 这也是重点
  • binlog
-rw-r-----  1 mysql mysql        177 May  3 11:23 tjtx-126-164.000001
-rw-r-----  1 mysql mysql 1074589597 May  3 11:29 tjtx-126-164.000002
-rw-r-----  1 mysql mysql 1074589060 May  3 11:30 tjtx-126-164.000003
-rw-r-----  1 mysql mysql 1074589063 May  3 11:31 tjtx-126-164.000004
-rw-r-----  1 mysql mysql 1074589065 May  3 11:32 tjtx-126-164.000005
-rw-r-----  1 mysql mysql 1074589051 May  3 11:33 tjtx-126-164.000006
-rw-r-----  1 mysql mysql 1074589045 May  3 11:33 tjtx-126-164.000007
-rw-r-----  1 mysql mysql 1074589047 May  3 11:34 tjtx-126-164.000008
-rw-r-----  1 mysql mysql 1074589050 May  3 11:35 tjtx-126-164.000009
-rw-r-----  1 mysql mysql 1074589052 May  3 11:36 tjtx-126-164.000010
-rw-r-----  1 mysql mysql 1074589062 May  3 11:37 tjtx-126-164.000011
-rw-r-----  1 mysql mysql 1074589068 May  3 11:37 tjtx-126-164.000012
-rw-r-----  1 mysql mysql 1074589045 May  3 11:38 tjtx-126-164.000013
-rw-r-----  1 mysql mysql 1074589038 May  3 11:39 tjtx-126-164.000014
-rw-r-----  1 mysql mysql 1074589055 May  3 11:40 tjtx-126-164.000015
-rw-r-----  1 mysql mysql 1074589050 May  3 11:41 tjtx-126-164.000016
-rw-r-----  1 mysql mysql 1074589063 May  3 11:41 tjtx-126-164.000017
-rw-r-----  1 mysql mysql 1074589055 May  3 11:42 tjtx-126-164.000018
-rw-r-----  1 mysql mysql 1074589048 May  3 11:43 tjtx-126-164.000019
-rw-r-----  1 mysql mysql 1074515950 May  3 11:45 tjtx-126-164.000020
-rw-r-----  1 mysql mysql 1074589069 May  3 11:46 tjtx-126-164.000021
-rw-r-----  1 mysql mysql 1074589051 May  3 11:47 tjtx-126-164.000022
-rw-r-----  1 mysql mysql 1074589063 May  3 11:47 tjtx-126-164.000023
-rw-r-----  1 mysql mysql 1074589051 May  3 11:48 tjtx-126-164.000024
-rw-r-----  1 mysql mysql  321034919 May  3 13:53 tjtx-126-164.000025
-rw-r-----  1 mysql mysql        204 May  3 13:53 tjtx-126-164.000026
-rw-r-----  1 mysql mysql        204 May  3 13:53 tjtx-126-164.000027
-rw-r-----  1 mysql mysql       1092 May  3 13:55 tjtx-126-164.000028
-rw-r-----  1 mysql mysql        194 May  3 13:55 tjtx-126-164.000029

tjtx-126-164.000001 ~ tjtx-126-164.000028

    Previous-GTIDs
    # [empty]


tjtx-126-164.000029

    #180503 13:55:05 server id 1261261646  end_log_pos 194 CRC32 0xb77b80b7     Previous-GTIDs
    # 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-3
  • 测试开始
<master>

dba:lc> purge binary logs to 'tjtx-126-164.000005';
Query OK, 0 rows affected (1 min 14.41 sec)   --执行时间竟然长达一分钟

dba:lc> insert into t select 300;  --master的事务卡住
Query OK, 1 row affected (1 min 9.42 sec)
Records: 1  Duplicates: 0  Warnings: 0

  • strace跟踪
在从头到尾遍历binlog  ,从而再次验证了我们之前的算法理论。


63639 14:07:50.394945 read(55, "05488963387-39206410793-66801786"..., 8192) = 8192 <0.000011>
63639 14:07:50.395005 read(55, "-66498258471-55447794725-7620591"..., 8192) = 8192 <0.000010>
63639 14:07:50.395065 read(55, "7;10709822844-35491948145-283531"..., 8192) = 8192 <0.000012>
63639 14:07:50.395129 read(55, "17-05336385032;74931753923-32217"..., 8192) = 8192 <0.000011>
63639 14:07:50.395191 read(55, "053-81565945575-96536403914;8342"..., 8192) = 8192 <0.000011>
63639 14:07:50.395250 read(55, "7139-77543559499-90858749831-907"..., 8192) = 8192 <0.000010>
63639 14:07:50.395310 read(55, "07981-10898305107-65423962210-93"..., 8192) = 8192 <0.000011>
63639 14:07:50.395371 read(55, "009985-68038808770-60998915978-7"..., 8192) = 8192 <0.000010>
63639 14:07:50.395430 read(55, "3665266-98504623794-11513728759-"..., 8192) = 8192 <0.000011>
63639 14:07:50.395491 read(55, "54495717-21332716078-74081433759"..., 8192) = 8192 <0.000010>
63639 14:07:50.395550 read(55, "873221923-40252274459-8633934300"..., 8192) = 8192 <0.000010>
63639 14:07:50.395610 read(55, "2609904861-91693621073-471178324"..., 8192) = 8192 <0.000010>
63639 14:07:50.125372 open("/data/mysql.bin/tjtx-126-164.~rec~", O_RDWR|O_CREAT, 0640) = 53 <0.000039>
63639 14:07:50.125769 open("/data/mysql.bin/tjtx-126-164.index_crash_safe", O_RDWR|O_CREAT, 0640) = 55 <0.000031>
63639 14:07:50.126150 open("/data/mysql.bin/tjtx-126-164.index", O_RDWR|O_CREAT, 0640) = 3 <0.000013>

。。。。。。。。。。。。。。。。。
63639 14:07:50.126554 open("/data/mysql.bin/tjtx-126-164.000005", O_RDONLY) = 55 <0.000012>
63639 14:07:53.857069 open("/data/mysql.bin/tjtx-126-164.000006", O_RDONLY) = 55 <0.000018>
63639 14:07:57.516826 open("/data/mysql.bin/tjtx-126-164.000007", O_RDONLY) = 55 <0.000016>
63639 14:08:01.169413 open("/data/mysql.bin/tjtx-126-164.000008", O_RDONLY) = 55 <0.000018>
63639 14:08:04.815608 open("/data/mysql.bin/tjtx-126-164.000009", O_RDONLY) = 55 <0.000015>
63639 14:08:08.473808 open("/data/mysql.bin/tjtx-126-164.000010", O_RDONLY) = 55 <0.000015>
63639 14:08:12.449964 open("/data/mysql.bin/tjtx-126-164.000011", O_RDONLY) = 55 <0.000018>
63639 14:08:16.251054 open("/data/mysql.bin/tjtx-126-164.000012", O_RDONLY) = 55 <0.000019>
63639 14:08:19.686003 open("/data/mysql.bin/tjtx-126-164.000013", O_RDONLY) = 55 <0.000015>
63639 14:08:23.341291 open("/data/mysql.bin/tjtx-126-164.000014", O_RDONLY) = 55 <0.000017>
63639 14:08:27.014210 open("/data/mysql.bin/tjtx-126-164.000015", O_RDONLY) = 55 <0.000016>
63639 14:08:30.625242 open("/data/mysql.bin/tjtx-126-164.000016", O_RDONLY) = 55 <0.000016>
63639 14:08:34.192385 open("/data/mysql.bin/tjtx-126-164.000017", O_RDONLY) = 55 <0.000015>
63639 14:08:37.862750 open("/data/mysql.bin/tjtx-126-164.000018", O_RDONLY) = 55 <0.000016>
63639 14:08:41.533869 open("/data/mysql.bin/tjtx-126-164.000019", O_RDONLY) = 55 <0.000016>
63639 14:08:45.202949 open("/data/mysql.bin/tjtx-126-164.000020", O_RDONLY) = 55 <0.000017>
63639 14:08:48.792088 open("/data/mysql.bin/tjtx-126-164.000021", O_RDONLY) = 55 <0.000017>
63639 14:08:52.266700 open("/data/mysql.bin/tjtx-126-164.000022", O_RDONLY) = 55 <0.000017>
63639 14:08:55.932879 open("/data/mysql.bin/tjtx-126-164.000023", O_RDONLY) = 55 <0.000017>
63639 14:08:59.594761 open("/data/mysql.bin/tjtx-126-164.000024", O_RDONLY) = 55 <0.000015>
63639 14:09:03.256451 open("/data/mysql.bin/tjtx-126-164.000025", O_RDONLY) = 55 <0.000015>
63639 14:09:04.349108 open("/data/mysql.bin/tjtx-126-164.000026", O_RDONLY) = 55 <0.000014>
63639 14:09:04.349280 open("/data/mysql.bin/tjtx-126-164.000027", O_RDONLY) = 55 <0.000010>
63639 14:09:04.349434 open("/data/mysql.bin/tjtx-126-164.000028", O_RDONLY) = 55 <0.000010>


查看fd=55的句柄:

[root@tjtx-126-164 tmp]# ll /proc/62382/fd | grep 55
lr-x------ 1 root root 64 May  3 14:17 55 -> /data/mysql.bin/tjtx-126-164.000009
[root@tjtx-126-164 tmp]# ll /proc/62382/fd | grep 55
lr-x------ 1 root root 64 May  3 14:17 55 -> /data/mysql.bin/tjtx-126-164.000010
[root@tjtx-126-164 tmp]# ll /proc/62382/fd | grep 55
lr-x------ 1 root root 64 May  3 14:17 55 -> /data/mysql.bin/tjtx-126-164.000010
[root@tjtx-126-164 tmp]# ll /proc/62382/fd | grep 55
lr-x------ 1 root root 64 May  3 14:17 55 -> /data/mysql.bin/tjtx-126-164.000010



测试二

  • 环境

    MySQL5.7.13
    binlog_gtid_simple_recovery = true
    non-GTID -> GTID 后,purge binary logs
  • 模拟开始
dba:(none)> purge binary logs to 'tjtx-126-164.000007';
Query OK, 0 rows affected (4.06 sec)  --非常快


dba:(none)> show global variables like '%gtid%';
+----------------------------------+------------------------------------------+
| Variable_name                    | Value                                    |
+----------------------------------+------------------------------------------+
| binlog_gtid_simple_recovery      | ON                                       |
| enforce_gtid_consistency         | ON                                       |
| gtid_executed                    | 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-7 |
| gtid_executed_compression_period | 1000                                     |
| gtid_mode                        | ON                                       |
| gtid_owned                       |                                          |
| gtid_purged                      |                                          |
| session_track_gtids              | OFF                                      |
+----------------------------------+------------------------------------------+
8 rows in set (0.00 sec)

strace分析:只读取了oldest的binlog 文件

115529 14:31:31.096480 open("/data/mysql.bin/tjtx-126-164.~rec~", O_RDWR|O_CREAT, 0640) = 51 <0.000031>
115529 14:31:31.096777 open("/data/mysql.bin/tjtx-126-164.index_crash_safe", O_RDWR|O_CREAT, 0640) = 52 <0.000029>
115529 14:31:31.097111 open("/data/mysql.bin/tjtx-126-164.index", O_RDWR|O_CREAT, 0640) = 3 <0.000023>
115529 14:31:31.097502 open("/data/mysql.bin/tjtx-126-164.000007", O_RDONLY) = 52 <0.000012>


dba:(none)> purge binary logs to 'tjtx-126-164.000029';
Query OK, 0 rows affected (0.00 sec)

dba:(none)> show global variables like '%gtid%';
+----------------------------------+------------------------------------------+
| Variable_name                    | Value                                    |
+----------------------------------+------------------------------------------+
| binlog_gtid_simple_recovery      | ON                                       |
| enforce_gtid_consistency         | ON                                       |
| gtid_executed                    | 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-7 |
| gtid_executed_compression_period | 1000                                     |
| gtid_mode                        | ON                                       |
| gtid_owned                       |                                          |
| gtid_purged                      | 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-3 |  --直到000029这个binlog文件读取,才能初始化gitid_purged值,否则为空
| session_track_gtids              | OFF                                      |
+----------------------------------+------------------------------------------+
8 rows in set (0.00 sec)



算法总结

1. MySQL重启
    当binlog_gtid_simple_recovery=FALSE 时:
        为了初始化 gtid_executed,算法是: 从newest_binlog -> oldest_binlog 方向遍历读取,如果发现有Previous_gtids_log_event , 那么就停止遍历。
        为了初始化 gtid_purged,算法是:   从oldest_binlog -> newest_binlog 方向遍历读取, 如果发现有Previous_gtids_log_event(not empty)或者 至少有一个Gtid_log_event的文件,那么就停止遍历
    当binlog_gtid_simple_recovery=TRUE 时:
           为了初始化 gtid_executed , 算法是: 只需要读取newest_binlog。 如果没有,则为空
        为了初始化 gtid_purged, 算法是: 只需要读取oldest_binlog。如果没有,则为空

2. binlog rotate(expire_logs_day , purge binary logs to '' 等)
    当binlog_gtid_simple_recovery=FALSE 时:
        为了初始化 gtid_purged , 从oldest_binlog -> newest_binlog 方向遍历读取, 如果发现有Previous_gtids_log_event(not empty)或者 至少有一个Gtid_log_event的文件,那么就停止遍历
    当binlog_gtid_simple_recovery=TRUE 时:
        为了初始化 gtid_purged, 算法是: 只需要读取oldest_binlog。 如果没有,则为空

需要注意的点

  • 在线GTID升级的时候,binlog_gtid_simple_recovery = TRUE 必须打开,否则在binlog 删除的时候,会发生阻塞状况
  • 在线GTID升级的时候,尽量将非GTID的binlog备份好,然后删除掉,以免出现莫名其妙的错误
相关实践学习
每个IT人都想学的“Web应用上云经典架构”实战
本实验从Web应用上云这个最基本的、最普遍的需求出发,帮助IT从业者们通过“阿里云Web应用上云解决方案”,了解一个企业级Web应用上云的常见架构,了解如何构建一个高可用、可扩展的企业级应用架构。
MySQL数据库入门学习
本课程通过最流行的开源数据库MySQL带你了解数据库的世界。 &nbsp; 相关的阿里云产品:云数据库RDS MySQL 版 阿里云关系型数据库RDS(Relational Database Service)是一种稳定可靠、可弹性伸缩的在线数据库服务,提供容灾、备份、恢复、迁移等方面的全套解决方案,彻底解决数据库运维的烦恼。 了解产品详情:&nbsp;https://www.aliyun.com/product/rds/mysql&nbsp;
目录
相关文章
|
人工智能 运维 关系型数据库
数据库运维:mysql 数据库迁移方法-mysqldump
本文介绍了MySQL数据库迁移的方法与技巧,重点探讨了数据量大小对迁移方式的影响。对于10GB以下的小型数据库,推荐使用mysqldump进行逻辑导出和source导入;10GB以上可考虑mydumper与myloader工具;100GB以上则建议物理迁移。文中还提供了统计数据库及表空间大小的SQL语句,并讲解了如何使用mysqldump导出存储过程、函数和数据结构。通过结合实际应用场景选择合适的工具与方法,可实现高效的数据迁移。
1790 1
|
11月前
|
SQL 运维 关系型数据库
深入探讨MySQL的二进制日志(binlog)选项
总结而言,对MySQL binlogs深度理解并妥善配置对数据库运维管理至关重要;它不仅关系到系统性能优化也是实现高可靠性架构设计必须考虑因素之一。通过精心规划与周密部署可以使得该机能充分发挥作用而避免潜在风险带来影响。
355 6
|
12月前
|
存储 SQL 关系型数据库
MySQL中binlog、redolog与undolog的不同之处解析
每个都扮演回答回溯与错误修正机构角色: BinLog像历史记载员详细记载每件大大小小事件; RedoLog则像紧急救援队伍遇见突發情況追踪最后活动轨迹尽力补救; UndoLog就类似时间机器可倒带历史让一切归位原始样貌同时兼具平行宇宙观察能让多人同时看见各自期望看见历程而互不干扰.
613 9
|
存储 SQL 关系型数据库
MySQL的Redo Log与Binlog机制对照分析
通过合理的配置和细致的管理,这两种日志机制相互配合,能够有效地提升MySQL数据库的可靠性和稳定性。
397 10
|
SQL 监控 关系型数据库
MySQL日志分析:binlog、redolog、undolog三大日志的深度探讨。
数据库管理其实和写小说一样,需要规划,需要修订,也需要有能力回滚。理解这些日志的作用与优化,就像把握写作工具的使用与运用,为我们的数据库保驾护航。
984 23
|
SQL 运维 关系型数据库
MySQL Binlog 日志查看方法及查看内容解析
本文介绍了 MySQL 的 Binlog(二进制日志)功能及其使用方法。Binlog 记录了数据库的所有数据变更操作,如 INSERT、UPDATE 和 DELETE,对数据恢复、主从复制和审计至关重要。文章详细说明了如何开启 Binlog 功能、查看当前日志文件及内容,并解析了常见的事件类型,包括 Format_desc、Query、Table_map、Write_rows、Update_rows 和 Delete_rows 等,帮助用户掌握数据库变化历史,提升维护和排障能力。
|
11月前
|
缓存 关系型数据库 BI
使用MYSQL Report分析数据库性能(下)
使用MYSQL Report分析数据库性能
628 158
|
11月前
|
关系型数据库 MySQL 数据库
自建数据库如何迁移至RDS MySQL实例
数据库迁移是一项复杂且耗时的工程,需考虑数据安全、完整性及业务中断影响。使用阿里云数据传输服务DTS,可快速、平滑完成迁移任务,将应用停机时间降至分钟级。您还可通过全量备份自建数据库并恢复至RDS MySQL实例,实现间接迁移上云。
|
11月前
|
关系型数据库 MySQL 数据库
阿里云数据库RDS费用价格:MySQL、SQL Server、PostgreSQL和MariaDB引擎收费标准
阿里云RDS数据库支持MySQL、SQL Server、PostgreSQL、MariaDB,多种引擎优惠上线!MySQL倚天版88元/年,SQL Server 2核4G仅299元/年,PostgreSQL 227元/年起。高可用、可弹性伸缩,安全稳定。详情见官网活动页。
1673 152
|
11月前
|
关系型数据库 MySQL 数据库
阿里云数据库RDS支持MySQL、SQL Server、PostgreSQL和MariaDB引擎
阿里云数据库RDS支持MySQL、SQL Server、PostgreSQL和MariaDB引擎,提供高性价比、稳定安全的云数据库服务,适用于多种行业与业务场景。
1125 156

推荐镜像

更多