RMQ 4.2.0-4.5.2[超时清除队列]

1、硬件配置

8C - 32G - ESSD200G

2、broker的配置：

brokerClusterName= brokerName= brokerId=0 namesrvAddr= listenPort=10911 fileReservedTime=48 diskMaxUsedSpaceRatio=95 storePathRootDir=/usr/local/rocketmq/store storePathCommitLog=/usr/local/rocketmq/store/commitlog storePathConsumerQueue=/usr/local/rocketmq/consumequeue brokerRole=SYNC_MASTER flushDiskType=ASYNC_FLUSH sendMessageThreadPoolNums=64 useReentrantLockWhenPutMessage=true

brokerRole=SYNC_MASTER flushDiskType=ASYNC_FLUSH sendMessageThreadPoolNums=64 useReentrantLockWhenPutMessage=true

3、问题现象：

A lot of flow control log information often occurs，By tracking the code, it is found that the synchronous replication between the master node and the slave node has timed out： do sync transfer other node, wait return, but failed, topic putMessage not in lock eclipse time(ms)=6194

log warn 1

org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2 DESC: [TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: 209ms, size of queue: 56

log warn 2

2019-11-09 15:46:08 ERROR SendMessageThread_26 - do sync transfer other node, wait return, but failed, topic: FUND_PENDING_CLEARING_MESSAGE_TOPIC tags: null client address: 10.32.0.157 2019-11-09 15:46:08 WARN SendMessageThread_26 - putMessage not in lock eclipse time(ms)=6194, bodyLength=2062 2019-11-09 15:46:08 ERROR SendMessageThread_57 - do sync transfer other node, wait return, but failed, topic: FUND_PENDING_CLEARING_MESSAGE_TOPIC tags: null client address: 10.32.0.157 2019-11-09 15:46:08 WARN SendMessageThread_57 - putMessage not in lock eclipse time(ms)=6194, bodyLength=2056

log warn 3 2019-11-08 08:21:06 WARN SendMessageThread_42 - putMessage not in lock eclipse time(ms)=3804, bodyLength=27 2019-11-08 08:21:06 WARN SendMessageThread_30 - putMessage not in lock eclipse time(ms)=3782, bodyLength=108 2019-11-08 08:21:06 WARN SendMessageThread_7 - putMessage not in lock eclipse time(ms)=3780, bodyLength=108 2019-11-08 08:21:06 WARN SendMessageThread_14 - putMessage not in lock eclipse time(ms)=3780, bodyLength=108 2019-11-08 08:21:06 WARN SendMessageThread_32 - putMessage not in lock eclipse time(ms)=3779, bodyLength=108 2019-11-08 08:21:06 WARN SendMessageThread_21 - putMessage not in lock eclipse time(ms)=3758, bodyLength=78 2019-11-08 08:21:06 WARN SendMessageThread_46 - putMessage not in lock eclipse time(ms)=3382, bodyLength=78 2019-11-08 08:21:06 WARN SendMessageThread_5 - putMessage not in lock eclipse time(ms)=3382, bodyLength=108 2019-11-08 08:21:06 WARN SendMessageThread_41 - putMessage not in lock eclipse time(ms)=3382, bodyLength=1641 2019-11-08 08:21:06 WARN SendMessageThread_16 - putMessage not in lock eclipse time(ms)=2595, bodyLength=2440 2019-11-08 08:21:06 WARN SendMessageThread_26 - putMessage not in lock eclipse time(ms)=3089, bodyLength=1601 2019-11-08 08:21:06 WARN SendMessageThread_4 - putMessage not in lock eclipse time(ms)=2353, bodyLength=2051 2019-11-08 08:21:06 WARN SendMessageThread_9 - putMessage not in lock eclipse time(ms)=2724, bodyLength=2414 2019-11-08 08:21:06 WARN SendMessageThread_34 - putMessage not in lock eclipse time(ms)=2351, bodyLength=2057 2019-11-08 08:21:06 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms)=3194, bodyLength=80 2019-11-08 08:21:06 WARN SendMessageThread_27 - putMessage not in lock eclipse time(ms)=2393, bodyLength=764 2019-11-08 08:21:06 WARN SendMessageThread_43 - putMessage not in lock eclipse time(ms)=2355, bodyLength=1988 2019-11-08 08:21:06 WARN SendMessageThread_24 - putMessage not in lock eclipse time(ms)=2120, bodyLength=3393 2019-11-08 08:21:06 WARN SendMessageThread_1 - putMessage not in lock eclipse time(ms)=2718, bodyLength=3097 2019-11-08 08:21:06 WARN SendMessageThread_29 - putMessage not in lock eclipse time(ms)=2727, bodyLength=2063 2019-11-08 08:21:06 WARN SendMessageThread_11 - putMessage not in lock eclipse time(ms)=2349, bodyLength=2063 2019-11-08 08:21:06 WARN SendMessageThread_38 - putMessage not in lock eclipse time(ms)=2358, bodyLength=1970 2019-11-08 08:21:06 WARN SendMessageThread_35 - putMessage not in lock eclipse time(ms)=1589, bodyLength=2042 2019-11-08 08:21:06 WARN SendMessageThread_22 - putMessage not in lock eclipse time(ms)=2511, bodyLength=1901 2019-11-08 08:21:06 WARN SendMessageThread_44 - putMessage not in lock eclipse time(ms)=2288, bodyLength=65 2019-11-08 08:21:06 WARN SendMessageThread_23 - putMessage not in lock eclipse time(ms)=1536, bodyLength=2057 2019-11-08 08:21:06 WARN SendMessageThread_28 - putMessage not in lock eclipse time(ms)=1908, bodyLength=1658 2019-11-08 08:21:06 WARN SendMessageThread_6 - putMessage not in lock eclipse time(ms)=2090, bodyLength=3512 2019-11-08 08:21:06 WARN SendMessageThread_13 - putMessage not in lock eclipse time(ms)=1480, bodyLength=108 2019-11-08 08:21:06 WARN SendMessageThread_8 - putMessage not in lock eclipse time(ms)=2095, bodyLength=3365 2019-11-08 08:21:06 WARN SendMessageThread_15 - putMessage not in lock eclipse time(ms)=2727, bodyLength=2051 2019-11-08 08:21:06 WARN SendMessageThread_48 - putMessage not in lock eclipse time(ms)=1774, bodyLength=2066 2019-11-08 08:21:06 WARN SendMessageThread_33 - putMessage not in lock eclipse time(ms)=1462, bodyLength=872 2019-11-08 08:21:06 WARN SendMessageThread_47 - putMessage not in lock eclipse time(ms)=1474, bodyLength=108 2019-11-08 08:21:06 WARN SendMessageThread_37 - putMessage not in lock eclipse time(ms)=2355, bodyLength=1284 2019-11-08 08:21:06 WARN SendMessageThread_40 - putMessage not in lock eclipse time(ms)=1589, bodyLength=2048 2019-11-08 08:21:06 WARN SendMessageThread_17 - putMessage not in lock eclipse time(ms)=1476, bodyLength=27 2019-11-08 08:21:06 WARN SendMessageThread_36 - putMessage not in lock eclipse time(ms)=3382, bodyLength=80 2019-11-08 08:21:06 WARN SendMessageThread_19 - putMessage not in lock eclipse time(ms)=2353, bodyLength=1268 2019-11-08 08:21:06 WARN SendMessageThread_45 - putMessage not in lock eclipse time(ms)=1562, bodyLength=27 2019-11-08 08:21:06 WARN SendMessageThread_10 - putMessage not in lock eclipse time(ms)=1535, bodyLength=2066 2019-11-08 08:21:06 WARN SendMessageThread_31 - putMessage not in lock eclipse time(ms)=1520, bodyLength=1949 2019-11-08 08:21:06 WARN SendMessageThread_20 - putMessage not in lock eclipse time(ms)=1534, bodyLength=2063 2019-11-08 08:21:06 WARN SendMessageThread_25 - putMessage not in lock eclipse time(ms)=1611, bodyLength=485 2019-11-08 08:21:06 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms)=1476, bodyLength=108 2019-11-08 08:21:06 WARN SendMessageThread_39 - putMessage not in lock eclipse time(ms)=1493, bodyLength=3469 2019-11-08 08:21:06 WARN SendMessageThread_12 - putMessage not in lock eclipse time(ms)=1374, bodyLength=1627 2019-11-08 08:21:06 WARN SendMessageThread_18 - putMessage not in lock eclipse time(ms)=1441, bodyLength=78

4、 如何排除和解决此类问题？

原提问者GitHub用户baihezhuo