MySQL死锁问题排查的case分析

本文涉及的产品
云数据库 RDS MySQL,集群版 2核4GB 100GB
推荐场景:
搭建个人博客
RDS MySQL Serverless 基础系列,0.5-2RCU 50GB
日志服务 SLS,月写入数据量 50GB 1个月
简介: MySQL死锁

查询死锁日志

查询死锁日志


登录MySQL,执行


show engine innodb status \G;

可以看到以下日志

LATEST DETECTED DEADLOCK ------------------------ 2019-07-29 11:40:16 0x2b05f879d700 *** (1) TRANSACTION: TRANSACTION 58675337, ACTIVE 0 sec fetching rows mysql tables in use 3, locked 3 LOCK WAIT 54 lock struct(s), heap size 8400, 4 row lock(s) MySQL thread id 5540968, OS thread handle 47304462178048, query id 2483784685 10.4.69.107 aaa_admin Searching rows for update update reimburse_expense         set status     = 'CLOSED',             updated_at = CURRENT_TIMESTAMP(6),             updated_by = 'UI181102VQXJFVG'         where ent_code = 'DTEC1809171U5IPQFR'           and status = 'PROCESSING'           and deleted = false           and reimburse_data_code = 'BX190725WY1I2V4' *** (1) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 2268 page no 9219 n bits 96 index PRIMARY of table `maycur-pro`.`reimburse_expense` trx id 58675337 lock_mode X locks rec but not gap waiting Record lock, heap no 29 PHYSICAL RECORD: n_fields 68; compact format; info bits 0  0: len 15; hex 46443139303732335857514532594f; asc FD190723XWQE2YO;;   *** (2) TRANSACTION: TRANSACTION 58675341, ACTIVE 0 sec fetching rows mysql tables in use 3, locked 3 52 lock struct(s), heap size 8400, 3 row lock(s) MySQL thread id 5542239, OS thread handle 47304643565312, query id 2483784712 10.4.69.107 cur_admin Searching rows for update update reimburse_expense         set status     = 'CLOSED',             updated_at = CURRENT_TIMESTAMP(6),             updated_by = 'UI181102VQXJFVG'         where ent_code = 'DTEC1809171U5IPQFR'           and status = 'PROCESSING'           and deleted = false           and reimburse_data_code = 'BX190725XGELH4W' *** (2) HOLDS THE LOCK(S): RECORD LOCKS space id 2268 page no 9219 n bits 96 index PRIMARY of table `xxx-pro`.`reimburse_expense` trx id 58675341 lock_mode X locks rec but not gap Record lock, heap no 29 PHYSICAL RECORD: n_fields 68; compact format; info bits 0  0: len 15; hex 46443139303732335857514532594f; asc FD190723XWQE2YO;;   *** (2) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 2268 page no 15201 n bits 328 index idx_reimburse_expense_status of table `maycur-pro`.`reimburse_expense` trx id 58675341 lock_mode X locks rec but not gap waiting Record lock, heap no 191 PHYSICAL RECORD: n_fields 3; compact format; info bits 0  0: len 18; hex 445445433138303931373155354950514652; asc DTEC1809171U5IPQFR;;  1: len 10; hex 50524f43455353494e47; asc PROCESSING;;  2: len 15; hex 46443139303732335857514532594f; asc FD190723XWQE2YO;; *** WE ROLL BACK TRANSACTION (2) ------------



MySQL记录的日志不完整,缺少部分可以推断出。


日志所示,


事务一(TRANSACTION 58675337)


等待RECORD LOCKS space id 2268 page no 9219 n bits 96 index PRIMARY of table xxx-pro.reimburse_expense trx id 58675337 lock_mode X locks rec but not gap waiting
Record lock, heap no 29 PHYSICAL RECORD: n_fields 68; compact format; info bits 0


事务二(TRANSACTION 58675341)


持有RECORD LOCKS space id 2268 page no 9219 n bits 96 index PRIMARY of table xxx-pro.reimburse_expensetrx id 58675341 lock_mode X locks rec but not gap Record lock


等待RECORD LOCKS space id 2268 page no 15201 n bits 328 index idx_reimburse_expense_status of table xxx-pro.reimburse_expensetrx id 58675341 lock_mode X locks rec but not gap waiting Record lock


根据据死锁条件可知,


事务一(TRANSACTION 58675337)


持有RECORD LOCKS space id 2268 page no 15201 n bits 328 index idx_reimburse_expense_status of table xxx-pro.reimburse_expensetrx id 58675341 lock_mode X locks rec but not gap waiting Record lock


以上信息表示


事务一持有idx_reimburse_expense_status锁,等了主键锁。


事务二持有主键锁,等待idx_reimburse_expense_status锁。



分析


reimburse_expense信息,省略了无关字段和索引


create table reimburse_expense_mock (     ent_code varchar(25) not null,     code varchar(25) not null,     reimburse_data_code varchar(25) null,     deleted tinyint(1) default 0 not null,     status varchar(16) default 'OPEN' not null,     primary key (code),     index fee_data_idx03(reimburse_data_code),     index idx_reimburse_expense_status(ent_code, status) );


引起死锁的SQL


# update1 update reimburse_expense set status     = 'CLOSED',     updated_at = CURRENT_TIMESTAMP(6),     updated_by = 'UI181102VQXJFVG' where ent_code = 'DTEC1809171U5IPQFR'   and status = 'PROCESSING'   and deleted = false   and reimburse_data_code = 'BX190725WY1I2V4'   # update2    update reimburse_expense set status     = 'CLOSED',     updated_at = CURRENT_TIMESTAMP(6),     updated_by = 'UI181102VQXJFVG' where ent_code = 'DTEC1809171U5IPQFR'   and status = 'PROCESSING'   and deleted = false   and reimburse_data_code = 'BX190725XGELH4W'



理论


RC隔离级别update执行过程


rc级别下,当update中使用了二级索引,执行过程:


  1. 扫描二级索引,在二级索引上加X锁
  2. 根据二级索引查询到的主键索引查询行数据,在主键索引上加X锁。验证
  3. 当行数据不符合查询条件,释放锁;符合,则更新。
  4. 当二级索引更新,会在未更新和更新后的索引上加X锁。验证



复现


猜测死锁产生的过程如下:


update执行过程:


第一步:update1扫描idx_reimburse_expense_status


第二步:update2扫描fee_data_idx03,这时update1和update2使用的是不同索引,不会阻塞。


第三步:update2使用第二步中扫描到的主键查询行,这个过程使用的是当前读,在主键上加X锁。


第四步:update1扫描的主键恰好是update2锁住的主键,update1使用该主键进行当前读就会发生阻塞。


第五步:update2使用主键更新status,status是二级索引idx_reimburse_expense_status一部分,二级索引发生更新,修改前后的索引都要加锁,而idx_reimburse_expense_status已经在第一步被update1加锁,update2阻塞,产生死锁。


具体步骤


模拟update,复现死锁。


Session1

Session2

start transaction;

start transaction;

1.1 select * from reimburse_expense_mock force index(idx_reimburse_expense_status) where ent_code = 'DTEC1809171U5IPQFR' and status = 'PROCESSING' and deleted = false and reimburse_data_code = 'BX190725WY1I2V4' for update ;

2.1 select from reimburse_expense_mock force index(fee_data_idx03) where ent_code = 'DTEC1809171U5IPQFR' and status = 'PROCESSING' and deleted = false and reimburse_data_code = 'BX190725XGELH4W' for update;
select from reimburse_expense_mock where code = 'FD190723XWQE2YO' for update ;

1.2 select * from reimburse_expense_mock where code = 'FD190723XWQE2YO' for update;(阻塞)

2.2 update reimburse_expense_mock force index (idx_reimburse_expense_status) set status = 'closing' where code='FD190723XWQE2YO';(死锁)



mysql死锁日志。如下

*** (1) TRANSACTION: TRANSACTION 12235990, ACTIVE 30 sec starting index read mysql tables in use 1, locked 1 LOCK WAIT 4 lock struct(s), heap size 1136, 38 row lock(s) MySQL thread id 227, OS thread handle 123145414512640, query id 38609 localhost 127.0.0.1 root statistics /* ApplicationName=IntelliJ IDEA 2019.1.2 */  select * from reimburse_expense_mock where code = 'FD190723XWQE2YO' for update *** (1) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 912 page no 3 n bits 104 index PRIMARY of table `eblank`.`reimburse_expense_mock` trx id 12235990 lock_mode X locks rec but not gap waiting Record lock, heap no 31 PHYSICAL RECORD: n_fields 69; compact format; info bits 0  0: len 15; hex 46443139303732335857514532594f; asc FD190723XWQE2YO;; *** (2) TRANSACTION: TRANSACTION 12235991, ACTIVE 27 sec starting index read mysql tables in use 1, locked 1 3 lock struct(s), heap size 1136, 2 row lock(s) MySQL thread id 228, OS thread handle 123145414234112, query id 38619 localhost 127.0.0.1 root Searching rows for update /* ApplicationName=IntelliJ IDEA 2019.1.2 */ update reimburse_expense_mock force index (idx_reimburse_expense_status) set status = 'closing' where ent_code = 'DTEC1809171U5IPQFR'   and status = 'PROCESSING'   and deleted = false   and code='FD190723XWQE2YO' *** (2) HOLDS THE LOCK(S): RECORD LOCKS space id 912 page no 3 n bits 104 index PRIMARY of table `eblank`.`reimburse_expense_mock` trx id 12235991 lock_mode X locks rec but not gap Record lock, heap no 31 PHYSICAL RECORD: n_fields 69; compact format; info bits 0  0: len 15; hex 46443139303732335857514532594f; asc FD190723XWQE2YO;; *** (2) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 912 page no 9 n bits 112 index idx_reimburse_expense_status of table `eblank`.`reimburse_expense_mock` trx id 12235991 lock_mode X locks rec but not gap waiting Record lock, heap no 20 PHYSICAL RECORD: n_fields 3; compact format; info bits 0  0: len 18; hex 445445433138303931373155354950514652; asc DTEC1809171U5IPQFR;;  1: len 10; hex 50524f43455353494e47; asc PROCESSING;;  2: len 15; hex 46443139303732335857514532594f; asc FD190723XWQE2YO;; *** WE ROLL BACK TRANSACTION (2)



和生产上的日志相比,加锁方式一致。



验证



二级索引查询


rc级别下update,会对二级索引加锁,再使用二级索引对应的主键进行当前读,锁住主键,读取后如果不符合查询条件,会释放。


使用以下方式证明。SQL1使用了二级索引idx_reimburse_expense_status,SQL2使用主键索引idx_fee_data_code。


# SQL1 update reimburse_expense_mock force index(idx_reimburse_expense_status) set status     = 'CLOSED',     updated_at = CURRENT_TIMESTAMP(6),     updated_by = 'UI181102VQXJFVG' where ent_code = 'DTEC1809171U5IPQFR'   and status = 'PROCESSING'   and deleted = false   and reimburse_data_code = 'BX190725WY1I2V4'; # SQL2 select * from reimburse_expense_mock where code = 'FD190723XWQE2YO' lock in share mode ;


  1. 先在事务一中执行SQL1,再在事务二中执行SQL2,不会发生阻塞。
  2. 先在事务一中执行SQL2,再在事务二中执行SQL1,SQL1发生阻塞。



二级索引更新


当二级索引更新,会在未更新和更新后的索引上加X锁。


# SQL1 update reimburse_expense_mock set status = 'closing' where code='FD190723XWQE2YO'; # SQL2 select * from reimburse_expense_mock force index(idx_reimburse_expense_status) where ent_code = 'DTEC1809171U5IPQFR'   and status = 'processing'     for update;      # SQL3 select * from reimburse_expense_mock force index(idx_reimburse_expense_status) where ent_code = 'DTEC1809171U5IPQFR'   and status = 'closing'     for update;



二级索引更新时,未更新的索引上会加锁。


Session1

Session2

update reimburse_expense_mock
set status = 'closing'
where code='FD190723XWQE2YO';

select *
from reimburse_expense_mock force index(idx_reimburse_expense_status)
where ent_code = 'DTEC1809171U5IPQFR'
and status = 'processing'
for update;(阻塞)



查看lock状态。


select lock_mode, lock_type, lock_table,lock_index, lock_data from information_schema.INNODB_LOCKS;

image.png



可以看出锁住了未更新的数据。



二级索引更新时,更新后的索引上会加锁。


Session1

Session2

update reimburse_expense_mock
set status = 'closing'
where code='FD190723XWQE2YO';

select * from reimburse_expense_mock force index(idx_reimburse_expense_status) where ent_code = 'DTEC1809171U5IPQFR' and status = 'closing' for update;(阻塞)



lock状态

image.png

相关实践学习
如何在云端创建MySQL数据库
开始实验后,系统会自动创建一台自建MySQL的 源数据库 ECS 实例和一台 目标数据库 RDS。
全面了解阿里云能为你做什么
阿里云在全球各地部署高效节能的绿色数据中心,利用清洁计算为万物互联的新世界提供源源不断的能源动力,目前开服的区域包括中国(华北、华东、华南、香港)、新加坡、美国(美东、美西)、欧洲、中东、澳大利亚、日本。目前阿里云的产品涵盖弹性计算、数据库、存储与CDN、分析与搜索、云通信、网络、管理与监控、应用服务、互联网中间件、移动服务、视频服务等。通过本课程,来了解阿里云能够为你的业务带来哪些帮助     相关的阿里云产品:云服务器ECS 云服务器 ECS(Elastic Compute Service)是一种弹性可伸缩的计算服务,助您降低 IT 成本,提升运维效率,使您更专注于核心业务创新。产品详情: https://www.aliyun.com/product/ecs
相关文章
|
11天前
|
SQL 监控 关系型数据库
深入解析MySQL死锁:原因、检测与解决方案
深入解析MySQL死锁:原因、检测与解决方案
|
18天前
|
存储 缓存 监控
MySQL 8.0中查询缓存的废弃与原因分析
MySQL 8.0中查询缓存的废弃与原因分析
35 1
|
23天前
|
SQL 关系型数据库 MySQL
MySQL数据库基础练习系列8、成绩录入与分析系统
MySQL数据库基础练习系列8、成绩录入与分析系统
14 1
|
23天前
|
存储 SQL 关系型数据库
【MySQL技术内幕】6.5-锁问题、阻塞、死锁、锁升级
【MySQL技术内幕】6.5-锁问题、阻塞、死锁、锁升级
47 2
|
25天前
|
SQL 关系型数据库 MySQL
基于Hive的天气情况大数据分析系统(通过hive进行大数据分析将分析的数据通过sqoop导入到mysql,通过Django基于mysql的数据做可视化)
基于Hive的天气情况大数据分析系统(通过hive进行大数据分析将分析的数据通过sqoop导入到mysql,通过Django基于mysql的数据做可视化)
|
8天前
|
关系型数据库 MySQL 测试技术
《阿里云产品四月刊》—瑶池数据库微课堂|RDS MySQL 经济版 vs 自建 MySQL 性能压测与性价比分析
阿里云瑶池数据库云原生化和一体化产品能力升级,多款产品更新迭代
|
11天前
|
关系型数据库 MySQL
MySQL中CASE WHEN用法总结
MySQL中CASE WHEN用法总结
|
11天前
|
关系型数据库 MySQL
mysql动态查列(case when then else end)
mysql动态查列(case when then else end)
9 0
|
23天前
|
SQL 关系型数据库 MySQL
MySQL——case when语句测试
MySQL——case when语句测试
12 0
|
1月前
|
存储 关系型数据库 MySQL
MySQL数据库——存储过程-if条件判断、参数、case(介绍、用法、案例)
MySQL数据库——存储过程-if条件判断、参数、case(介绍、用法、案例)
35 0