MySQL死锁问题排查的case分析

本文涉及的产品
RDS MySQL Serverless 基础系列,0.5-2RCU 50GB
RDS MySQL Serverless 高可用系列,价值2615元额度,1个月
简介: MySQL死锁

查询死锁日志

查询死锁日志


登录MySQL,执行


show engine innodb status \G;

可以看到以下日志

LATEST DETECTED DEADLOCK ------------------------ 2019-07-29 11:40:16 0x2b05f879d700 *** (1) TRANSACTION: TRANSACTION 58675337, ACTIVE 0 sec fetching rows mysql tables in use 3, locked 3 LOCK WAIT 54 lock struct(s), heap size 8400, 4 row lock(s) MySQL thread id 5540968, OS thread handle 47304462178048, query id 2483784685 10.4.69.107 aaa_admin Searching rows for update update reimburse_expense         set status     = 'CLOSED',             updated_at = CURRENT_TIMESTAMP(6),             updated_by = 'UI181102VQXJFVG'         where ent_code = 'DTEC1809171U5IPQFR'           and status = 'PROCESSING'           and deleted = false           and reimburse_data_code = 'BX190725WY1I2V4' *** (1) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 2268 page no 9219 n bits 96 index PRIMARY of table `maycur-pro`.`reimburse_expense` trx id 58675337 lock_mode X locks rec but not gap waiting Record lock, heap no 29 PHYSICAL RECORD: n_fields 68; compact format; info bits 0  0: len 15; hex 46443139303732335857514532594f; asc FD190723XWQE2YO;;   *** (2) TRANSACTION: TRANSACTION 58675341, ACTIVE 0 sec fetching rows mysql tables in use 3, locked 3 52 lock struct(s), heap size 8400, 3 row lock(s) MySQL thread id 5542239, OS thread handle 47304643565312, query id 2483784712 10.4.69.107 cur_admin Searching rows for update update reimburse_expense         set status     = 'CLOSED',             updated_at = CURRENT_TIMESTAMP(6),             updated_by = 'UI181102VQXJFVG'         where ent_code = 'DTEC1809171U5IPQFR'           and status = 'PROCESSING'           and deleted = false           and reimburse_data_code = 'BX190725XGELH4W' *** (2) HOLDS THE LOCK(S): RECORD LOCKS space id 2268 page no 9219 n bits 96 index PRIMARY of table `xxx-pro`.`reimburse_expense` trx id 58675341 lock_mode X locks rec but not gap Record lock, heap no 29 PHYSICAL RECORD: n_fields 68; compact format; info bits 0  0: len 15; hex 46443139303732335857514532594f; asc FD190723XWQE2YO;;   *** (2) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 2268 page no 15201 n bits 328 index idx_reimburse_expense_status of table `maycur-pro`.`reimburse_expense` trx id 58675341 lock_mode X locks rec but not gap waiting Record lock, heap no 191 PHYSICAL RECORD: n_fields 3; compact format; info bits 0  0: len 18; hex 445445433138303931373155354950514652; asc DTEC1809171U5IPQFR;;  1: len 10; hex 50524f43455353494e47; asc PROCESSING;;  2: len 15; hex 46443139303732335857514532594f; asc FD190723XWQE2YO;; *** WE ROLL BACK TRANSACTION (2) ------------



MySQL记录的日志不完整,缺少部分可以推断出。


日志所示,


事务一(TRANSACTION 58675337)


等待RECORD LOCKS space id 2268 page no 9219 n bits 96 index PRIMARY of table xxx-pro.reimburse_expense trx id 58675337 lock_mode X locks rec but not gap waiting
Record lock, heap no 29 PHYSICAL RECORD: n_fields 68; compact format; info bits 0


事务二(TRANSACTION 58675341)


持有RECORD LOCKS space id 2268 page no 9219 n bits 96 index PRIMARY of table xxx-pro.reimburse_expensetrx id 58675341 lock_mode X locks rec but not gap Record lock


等待RECORD LOCKS space id 2268 page no 15201 n bits 328 index idx_reimburse_expense_status of table xxx-pro.reimburse_expensetrx id 58675341 lock_mode X locks rec but not gap waiting Record lock


根据据死锁条件可知,


事务一(TRANSACTION 58675337)


持有RECORD LOCKS space id 2268 page no 15201 n bits 328 index idx_reimburse_expense_status of table xxx-pro.reimburse_expensetrx id 58675341 lock_mode X locks rec but not gap waiting Record lock


以上信息表示


事务一持有idx_reimburse_expense_status锁,等了主键锁。


事务二持有主键锁,等待idx_reimburse_expense_status锁。



分析


reimburse_expense信息,省略了无关字段和索引


create table reimburse_expense_mock (     ent_code varchar(25) not null,     code varchar(25) not null,     reimburse_data_code varchar(25) null,     deleted tinyint(1) default 0 not null,     status varchar(16) default 'OPEN' not null,     primary key (code),     index fee_data_idx03(reimburse_data_code),     index idx_reimburse_expense_status(ent_code, status) );


引起死锁的SQL


# update1 update reimburse_expense set status     = 'CLOSED',     updated_at = CURRENT_TIMESTAMP(6),     updated_by = 'UI181102VQXJFVG' where ent_code = 'DTEC1809171U5IPQFR'   and status = 'PROCESSING'   and deleted = false   and reimburse_data_code = 'BX190725WY1I2V4'   # update2    update reimburse_expense set status     = 'CLOSED',     updated_at = CURRENT_TIMESTAMP(6),     updated_by = 'UI181102VQXJFVG' where ent_code = 'DTEC1809171U5IPQFR'   and status = 'PROCESSING'   and deleted = false   and reimburse_data_code = 'BX190725XGELH4W'



理论


RC隔离级别update执行过程


rc级别下,当update中使用了二级索引,执行过程:


  1. 扫描二级索引,在二级索引上加X锁
  2. 根据二级索引查询到的主键索引查询行数据,在主键索引上加X锁。验证
  3. 当行数据不符合查询条件,释放锁;符合,则更新。
  4. 当二级索引更新,会在未更新和更新后的索引上加X锁。验证



复现


猜测死锁产生的过程如下:


update执行过程:


第一步:update1扫描idx_reimburse_expense_status


第二步:update2扫描fee_data_idx03,这时update1和update2使用的是不同索引,不会阻塞。


第三步:update2使用第二步中扫描到的主键查询行,这个过程使用的是当前读,在主键上加X锁。


第四步:update1扫描的主键恰好是update2锁住的主键,update1使用该主键进行当前读就会发生阻塞。


第五步:update2使用主键更新status,status是二级索引idx_reimburse_expense_status一部分,二级索引发生更新,修改前后的索引都要加锁,而idx_reimburse_expense_status已经在第一步被update1加锁,update2阻塞,产生死锁。


具体步骤


模拟update,复现死锁。


Session1

Session2

start transaction;

start transaction;

1.1 select * from reimburse_expense_mock force index(idx_reimburse_expense_status) where ent_code = 'DTEC1809171U5IPQFR' and status = 'PROCESSING' and deleted = false and reimburse_data_code = 'BX190725WY1I2V4' for update ;

2.1 select from reimburse_expense_mock force index(fee_data_idx03) where ent_code = 'DTEC1809171U5IPQFR' and status = 'PROCESSING' and deleted = false and reimburse_data_code = 'BX190725XGELH4W' for update;
select from reimburse_expense_mock where code = 'FD190723XWQE2YO' for update ;

1.2 select * from reimburse_expense_mock where code = 'FD190723XWQE2YO' for update;(阻塞)

2.2 update reimburse_expense_mock force index (idx_reimburse_expense_status) set status = 'closing' where code='FD190723XWQE2YO';(死锁)



mysql死锁日志。如下

*** (1) TRANSACTION: TRANSACTION 12235990, ACTIVE 30 sec starting index read mysql tables in use 1, locked 1 LOCK WAIT 4 lock struct(s), heap size 1136, 38 row lock(s) MySQL thread id 227, OS thread handle 123145414512640, query id 38609 localhost 127.0.0.1 root statistics /* ApplicationName=IntelliJ IDEA 2019.1.2 */  select * from reimburse_expense_mock where code = 'FD190723XWQE2YO' for update *** (1) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 912 page no 3 n bits 104 index PRIMARY of table `eblank`.`reimburse_expense_mock` trx id 12235990 lock_mode X locks rec but not gap waiting Record lock, heap no 31 PHYSICAL RECORD: n_fields 69; compact format; info bits 0  0: len 15; hex 46443139303732335857514532594f; asc FD190723XWQE2YO;; *** (2) TRANSACTION: TRANSACTION 12235991, ACTIVE 27 sec starting index read mysql tables in use 1, locked 1 3 lock struct(s), heap size 1136, 2 row lock(s) MySQL thread id 228, OS thread handle 123145414234112, query id 38619 localhost 127.0.0.1 root Searching rows for update /* ApplicationName=IntelliJ IDEA 2019.1.2 */ update reimburse_expense_mock force index (idx_reimburse_expense_status) set status = 'closing' where ent_code = 'DTEC1809171U5IPQFR'   and status = 'PROCESSING'   and deleted = false   and code='FD190723XWQE2YO' *** (2) HOLDS THE LOCK(S): RECORD LOCKS space id 912 page no 3 n bits 104 index PRIMARY of table `eblank`.`reimburse_expense_mock` trx id 12235991 lock_mode X locks rec but not gap Record lock, heap no 31 PHYSICAL RECORD: n_fields 69; compact format; info bits 0  0: len 15; hex 46443139303732335857514532594f; asc FD190723XWQE2YO;; *** (2) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 912 page no 9 n bits 112 index idx_reimburse_expense_status of table `eblank`.`reimburse_expense_mock` trx id 12235991 lock_mode X locks rec but not gap waiting Record lock, heap no 20 PHYSICAL RECORD: n_fields 3; compact format; info bits 0  0: len 18; hex 445445433138303931373155354950514652; asc DTEC1809171U5IPQFR;;  1: len 10; hex 50524f43455353494e47; asc PROCESSING;;  2: len 15; hex 46443139303732335857514532594f; asc FD190723XWQE2YO;; *** WE ROLL BACK TRANSACTION (2)



和生产上的日志相比,加锁方式一致。



验证



二级索引查询


rc级别下update,会对二级索引加锁,再使用二级索引对应的主键进行当前读,锁住主键,读取后如果不符合查询条件,会释放。


使用以下方式证明。SQL1使用了二级索引idx_reimburse_expense_status,SQL2使用主键索引idx_fee_data_code。


# SQL1 update reimburse_expense_mock force index(idx_reimburse_expense_status) set status     = 'CLOSED',     updated_at = CURRENT_TIMESTAMP(6),     updated_by = 'UI181102VQXJFVG' where ent_code = 'DTEC1809171U5IPQFR'   and status = 'PROCESSING'   and deleted = false   and reimburse_data_code = 'BX190725WY1I2V4'; # SQL2 select * from reimburse_expense_mock where code = 'FD190723XWQE2YO' lock in share mode ;


  1. 先在事务一中执行SQL1,再在事务二中执行SQL2,不会发生阻塞。
  2. 先在事务一中执行SQL2,再在事务二中执行SQL1,SQL1发生阻塞。



二级索引更新


当二级索引更新,会在未更新和更新后的索引上加X锁。


# SQL1 update reimburse_expense_mock set status = 'closing' where code='FD190723XWQE2YO'; # SQL2 select * from reimburse_expense_mock force index(idx_reimburse_expense_status) where ent_code = 'DTEC1809171U5IPQFR'   and status = 'processing'     for update;      # SQL3 select * from reimburse_expense_mock force index(idx_reimburse_expense_status) where ent_code = 'DTEC1809171U5IPQFR'   and status = 'closing'     for update;



二级索引更新时,未更新的索引上会加锁。


Session1

Session2

update reimburse_expense_mock
set status = 'closing'
where code='FD190723XWQE2YO';

select *
from reimburse_expense_mock force index(idx_reimburse_expense_status)
where ent_code = 'DTEC1809171U5IPQFR'
and status = 'processing'
for update;(阻塞)



查看lock状态。


select lock_mode, lock_type, lock_table,lock_index, lock_data from information_schema.INNODB_LOCKS;

image.png



可以看出锁住了未更新的数据。



二级索引更新时,更新后的索引上会加锁。


Session1

Session2

update reimburse_expense_mock
set status = 'closing'
where code='FD190723XWQE2YO';

select * from reimburse_expense_mock force index(idx_reimburse_expense_status) where ent_code = 'DTEC1809171U5IPQFR' and status = 'closing' for update;(阻塞)



lock状态

image.png

相关实践学习
基于CentOS快速搭建LAMP环境
本教程介绍如何搭建LAMP环境,其中LAMP分别代表Linux、Apache、MySQL和PHP。
全面了解阿里云能为你做什么
阿里云在全球各地部署高效节能的绿色数据中心,利用清洁计算为万物互联的新世界提供源源不断的能源动力,目前开服的区域包括中国(华北、华东、华南、香港)、新加坡、美国(美东、美西)、欧洲、中东、澳大利亚、日本。目前阿里云的产品涵盖弹性计算、数据库、存储与CDN、分析与搜索、云通信、网络、管理与监控、应用服务、互联网中间件、移动服务、视频服务等。通过本课程,来了解阿里云能够为你的业务带来哪些帮助     相关的阿里云产品:云服务器ECS 云服务器 ECS(Elastic Compute Service)是一种弹性可伸缩的计算服务,助您降低 IT 成本,提升运维效率,使您更专注于核心业务创新。产品详情: https://www.aliyun.com/product/ecs
相关文章
|
3天前
|
关系型数据库 MySQL 索引
mysql 分析5语句的优化--索引添加删除
mysql 分析5语句的优化--索引添加删除
17 0
|
3天前
|
关系型数据库 MySQL 数据挖掘
轻松入门MySQL:利用MySQL时间函数优化产品销售数据统计与分析(9)
轻松入门MySQL:利用MySQL时间函数优化产品销售数据统计与分析(9)
|
3天前
|
SQL 关系型数据库 MySQL
【MySQL技术专题】「问题实战系列」深入探索和分析MySQL数据库的数据备份和恢复实战开发指南(8.0版本升级篇)
【MySQL技术专题】「问题实战系列」深入探索和分析MySQL数据库的数据备份和恢复实战开发指南(8.0版本升级篇)
104 0
|
3天前
|
SQL 缓存 关系型数据库
mysql性能优化-慢查询分析、优化索引和配置
mysql性能优化-慢查询分析、优化索引和配置
112 1
|
3天前
|
缓存 关系型数据库 MySQL
MySQL 查询优化:提速查询效率的13大秘籍(索引设计、查询优化、缓存策略、子查询优化以及定期表分析和优化)(中)
MySQL 查询优化:提速查询效率的13大秘籍(索引设计、查询优化、缓存策略、子查询优化以及定期表分析和优化)(中)
|
3天前
|
存储 关系型数据库 MySQL
MySQL 8 索引原理详细分析
了解索引的详细原则,不仅有助于优化,能把索引搞清楚的,面试中优势也会很突显。 关于数据库优化的话题,V哥觉得还有很多地方可以聊,如果你有兴趣,欢迎关注一起讨论。
MySQL 8 索引原理详细分析
|
3天前
|
SQL 关系型数据库 MySQL
【mysql】mysq优化全方面分析
【mysql】mysq优化全方面分析
19 0
|
3天前
|
关系型数据库 MySQL 中间件
【MySQL实战笔记】07 | 行锁功过:怎么减少行锁对性能的影响?-02 死锁和死锁检测
【4月更文挑战第19天】在高并发环境下,死锁发生在多个线程间循环等待资源时,导致无限期等待。MySQL中,死锁可通过`innodb_lock_wait_timeout`参数设置超时或`innodb_deadlock_detect`开启死锁检测来解决。默认的50s超时可能不适用于在线服务,而频繁检测会消耗大量CPU。应对热点行更新引发的性能问题,可以暂时关闭死锁检测(风险是产生大量超时),控制并发度,或通过分散记录减少锁冲突,例如将数据分拆到多行以降低死锁概率。
25 1
|
3天前
|
SQL 关系型数据库 MySQL
【MySQL】慢SQL分析流程
【4月更文挑战第1天】【MySQL】慢SQL分析流程
|
3天前
|
关系型数据库 MySQL 数据库
docker MySQL删除数据库时的错误(errno: 39)
docker MySQL删除数据库时的错误(errno: 39)
24 0

推荐镜像

更多