Galera Cluster中节点异常宕机排查

背景

在Group Replication发布之前，MySQL官方复制有异步、半同步。当时弥补全同步的方案，大多数公司会选择Galera cluster，主要有percona server的PXC和MariaDB的MGC两种版本，而且都嵌入到各自的版本中。本文针对客户生产环境使用Galera Cluster（MGC）遇到的一则宕机案例

环境信息

• MariaDB 10.0.15
• redhat 6.5

日志信息

节点二（正常）日志：

190308 17:08:43 [Note] WSREP: Member 0.0 (node23) requested state transfer from '*any*'. Selected 1.0 (node144)(SYNCED) as donor.
190308 17:08:43 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 397258687)
190308 17:08:43 [Note] WSREP: IST request: a6befc67-f455-11e6-a8e6-fa93a785f2f6:397258655-397258656|tcp://21.244.57.46:4568
190308 17:08:43 [Note] WSREP: IST first seqno 397258656 not found from cache, falling back to SST
190308 17:08:43 [Warning] WSREP: SST request is null, SST canceled.

节点三（宕机）日志：

190308 17:08:43 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 397258687)
190308 17:08:43 [Note] WSREP: Requesting state transfer: success after 2 tries, donor: 1
190308 17:08:43 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): discarded 0 bytes
190308 17:08:43 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): found 1/31 locked buffers
190308 17:08:43 [Warning] WSREP: 1.0 (node144): State transfer to 0.0 (node23) failed: -125 (Operation canceled)
190308 17:08:43 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():723: Will never receive state. Need to abort.

总结：

• 节点二所在的MySQL实例作为节点三的donor；而且从节点二日志可以看出：事务a6befc67-f455-11e6-a8e6-fa93a785f2f6:397258655-397258656已经不在gcache中（丢失），从而导致节点三IST失败，只能重新启动节点三MySQL实例，通过SST来全量同步重新加入集群。

建议：

• 调大参数gcache.size的值，使得gcache中能够存储更多的事务