开发者社区 > 云原生 > 正文

关于[NOTIFYME]putMessage的问题锁定成本时间(毫秒)=4052

我在生产环境中使用过rocketmq。当用户向代理发送消息时,代理日志中有一些警告。导致消息发送失败。当它被写入页面缓存时,它看起来像是被阻止了。

请给我一些改进这个问题的建议。非常感谢。

在我们的应用场景中,我们使用的延迟消息比实时消息多得多。每当我们使用大量延迟消息时,就会出现这些异常。这有关系吗?

1、Tencent cloud linux 8 CPU. 48G MEM. SSD disk I have run broker after run os.sh. I can't set io scheduler to DEADLINE, because machine of cloud not support. default is none 2、broker.conf

fileReservedTime = 48 flushDiskType = ASYNC_FLUSH sendMessageThreadPoolNums=48 messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h 4h 6h 12h 1d 3h 5h 7h 8h 9h 10h 11h 13h 14h 15h 16h 17h 18h 19h 20h 21h 22h 23h 40m 50m waitTimeMillsInSendQueue=2000 sendThreadPoolQueueCapacity=50000

3、JAVAOPTS is default 4、TPS is less than 2000

一些日志:

2019-01-22 11:03:02 WARN SendMessageThread_19 - [NOTIFYME]putMessage in lock cost time(ms)=4127, bodyLength=410 AppendMessageResult=AppendMessageResult{status=PUT_OK, wroteOffset=3551463997150, wroteBytes=792, msgId='0A01003100002A9F0000033AE3BFFEDE', storeTimestamp=1548126178835, logicsOffset=422117414, pagecacheRT=4127, msgNum=1}

2019-01-22 11:03:02 WARN SendMessageThread_22 - putMessage not in lock eclipse time(ms)=4034, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_25 - putMessage not in lock eclipse time(ms)=4128, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_44 - putMessage not in lock eclipse time(ms)=4061, bodyLength=411 2019-01-22 11:03:02 WARN SendMessageThread_36 - putMessage not in lock eclipse time(ms)=4057, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_40 - putMessage not in lock eclipse time(ms)=4103, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_21 - putMessage not in lock eclipse time(ms)=4127, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_20 - putMessage not in lock eclipse time(ms)=4081, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_16 - putMessage not in lock eclipse time(ms)=4127, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_30 - putMessage not in lock eclipse time(ms)=4093, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_35 - putMessage not in lock eclipse time(ms)=4067, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_13 - putMessage not in lock eclipse time(ms)=4136, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_37 - putMessage not in lock eclipse time(ms)=3939, bodyLength=9 2019-01-22 11:03:02 WARN SendMessageThread_26 - putMessage not in lock eclipse time(ms)=4034, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_42 - putMessage not in lock eclipse time(ms)=4082, bodyLength=397 2019-01-22 11:03:02 WARN SendMessageThread_39 - putMessage not in lock eclipse time(ms)=4127, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_46 - putMessage not in lock eclipse time(ms)=4012, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_14 - putMessage not in lock eclipse time(ms)=4056, bodyLength=411 2019-01-22 11:03:02 WARN SendMessageThread_6 - putMessage not in lock eclipse time(ms)=4001, bodyLength=411 2019-01-22 11:03:02 WARN SendMessageThread_11 - putMessage not in lock eclipse time(ms)=4138, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_32 - putMessage not in lock eclipse time(ms)=4092, bodyLength=367 2019-01-22 11:03:02 WARN SendMessageThread_4 - putMessage not in lock eclipse time(ms)=4123, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_29 - putMessage not in lock eclipse time(ms)=4138, bodyLength=411 2019-01-22 11:03:02 WARN SendMessageThread_1 - putMessage not in lock eclipse time(ms)=4102, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_10 - putMessage not in lock eclipse time(ms)=4087, bodyLength=411 2019-01-22 11:03:02 WARN SendMessageThread_12 - putMessage not in lock eclipse time(ms)=4080, bodyLength=411 2019-01-22 11:03:02 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms)=4081, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_33 - putMessage not in lock eclipse time(ms)=4103, bodyLength=411 2019-01-22 11:03:02 WARN SendMessageThread_27 - putMessage not in lock eclipse time(ms)=3980, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_43 - putMessage not in lock eclipse time(ms)=3951, bodyLength=411 2019-01-22 11:03:02 WARN SendMessageThread_41 - putMessage not in lock eclipse time(ms)=4046, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_38 - putMessage not in lock eclipse time(ms)=3991, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_18 - putMessage not in lock eclipse time(ms)=4083, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms)=4122, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_7 - putMessage not in lock eclipse time(ms)=4120, bodyLength=367 2019-01-22 11:03:02 WARN SendMessageThread_15 - putMessage not in lock eclipse time(ms)=3980, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_34 - putMessage not in lock eclipse time(ms)=4122, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_8 - putMessage not in lock eclipse time(ms)=4077, bodyLength=411 2019-01-22 11:03:02 WARN SendMessageThread_28 - putMessage not in lock eclipse time(ms)=4074, bodyLength=411 2019-01-22 11:03:02 WARN SendMessageThread_17 - putMessage not in lock eclipse time(ms)=4024, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_31 - putMessage not in lock eclipse time(ms)=3962, bodyLength=410 2019-01-22 11:03:02 WARN ScheduleMessageTimerThread - putMessage not in lock eclipse time(ms)=4128, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_47 - putMessage not in lock eclipse time(ms)=4127, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_48 - putMessage not in lock eclipse time(ms)=4083, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_45 - putMessage not in lock eclipse time(ms)=4113, bodyLength=411 2019-01-22 11:03:02 WARN SendMessageThread_24 - putMessage not in lock eclipse time(ms)=3951, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_19 - putMessage not in lock eclipse time(ms)=4128, bodyLength=410 2019-01-22 11:03:02 WARN SendMessageThread_9 - putMessage not in lock eclipse time(ms)=3969, bodyLength=411 2019-01-22 11:03:02 WARN SendMessageThread_5 - putMessage not in lock eclipse time(ms)=4093, bodyLength=422 2019-01-22 11:03:02 WARN SendMessageThread_23 - putMessage not in lock eclipse time(ms)=3928, bodyLength=8

2019-01-25 12:24:07 WARN SendMessageThread_21 - putMessage not in lock eclipse time(ms)=537, bodyLength=206 2019-01-25 12:24:07 WARN SendMessageThread_5 - putMessage not in lock eclipse time(ms)=509, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_28 - putMessage not in lock eclipse time(ms)=656, bodyLength=221 2019-01-25 12:24:07 WARN SendMessageThread_35 - putMessage not in lock eclipse time(ms)=602, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_10 - putMessage not in lock eclipse time(ms)=685, bodyLength=438 2019-01-25 12:24:07 WARN SendMessageThread_14 - putMessage not in lock eclipse time(ms)=597, bodyLength=523 2019-01-25 12:24:07 WARN SendMessageThread_44 - putMessage not in lock eclipse time(ms)=719, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_16 - putMessage not in lock eclipse time(ms)=509, bodyLength=200 2019-01-25 12:24:07 WARN SendMessageThread_13 - putMessage not in lock eclipse time(ms)=603, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_9 - putMessage not in lock eclipse time(ms)=587, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_18 - putMessage not in lock eclipse time(ms)=521, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_23 - putMessage not in lock eclipse time(ms)=507, bodyLength=49 2019-01-25 12:24:07 WARN SendMessageThread_31 - putMessage not in lock eclipse time(ms)=546, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_32 - putMessage not in lock eclipse time(ms)=627, bodyLength=115 2019-01-25 12:24:07 WARN ScheduleMessageTimerThread - putMessage not in lock eclipse time(ms)=722, bodyLength=273 2019-01-25 12:24:07 WARN SendMessageThread_12 - putMessage not in lock eclipse time(ms)=714, bodyLength=566 2019-01-25 12:24:07 WARN SendMessageThread_41 - putMessage not in lock eclipse time(ms)=732, bodyLength=338 2019-01-25 12:24:07 WARN SendMessageThread_1 - putMessage not in lock eclipse time(ms)=563, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_15 - putMessage not in lock eclipse time(ms)=707, bodyLength=201 2019-01-25 12:24:07 WARN SendMessageThread_40 - putMessage not in lock eclipse time(ms)=563, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms)=603, bodyLength=238 2019-01-25 12:24:07 WARN SendMessageThread_17 - putMessage not in lock eclipse time(ms)=530, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_8 - putMessage not in lock eclipse time(ms)=736, bodyLength=408 2019-01-25 12:24:07 WARN SendMessageThread_11 - putMessage not in lock eclipse time(ms)=533, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_20 - putMessage not in lock eclipse time(ms)=688, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_29 - putMessage not in lock eclipse time(ms)=741, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms)=508, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_7 - putMessage not in lock eclipse time(ms)=707, bodyLength=74 2019-01-25 12:24:07 WARN SendMessageThread_46 - putMessage not in lock eclipse time(ms)=688, bodyLength=259 2019-01-25 12:24:07 WARN SendMessageThread_30 - putMessage not in lock eclipse time(ms)=680, bodyLength=229 2019-01-25 12:24:07 WARN SendMessageThread_36 - putMessage not in lock eclipse time(ms)=659, bodyLength=228 2019-01-25 12:24:07 WARN SendMessageThread_34 - putMessage not in lock eclipse time(ms)=551, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_39 - putMessage not in lock eclipse time(ms)=664, bodyLength=100 2019-01-25 12:24:07 WARN SendMessageThread_4 - putMessage not in lock eclipse time(ms)=749, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_33 - [NOTIFYME]putMessage in lock cost time(ms)=723, bodyLength=99 AppendMessageResult=AppendMessageResult{status=PUT_OK, wroteOffset=3637677915929, wroteBytes=398, msgId='0A01003100002A9F0000034EF67FFF19', storeTimestamp=1548390246329, logicsOffset=3777623, pagecacheRT=723, msgNum=1} 2019-01-25 12:24:07 WARN SendMessageThread_33 - putMessage not in lock eclipse time(ms)=768, bodyLength=99 2019-01-25 12:24:07 WARN SendMessageThread_42 - putMessage not in lock eclipse time(ms)=645, bodyLength=110 2019-01-25 12:24:07 WARN SendMessageThread_6 - putMessage not in lock eclipse time(ms)=580, bodyLength=200 2019-01-25 12:24:07 WARN SendMessageThread_43 - putMessage not in lock eclipse time(ms)=520, bodyLength=74 2019-01-25 12:24:07 WARN SendMessageThread_37 - putMessage not in lock eclipse time(ms)=617, bodyLength=74 2019-01-25 12:24:07 WARN SendMessageThread_27 - putMessage not in lock eclipse time(ms)=685, bodyLength=100 2019-01-25 12:24:07 WARN SendMessageThread_47 - putMessage not in lock eclipse time(ms)=627, bodyLength=559 2019-01-25 12:24:07 WARN SendMessageThread_25 - putMessage not in lock eclipse time(ms)=718, bodyLength=279 2019-01-25 12:24:07 WARN SendMessageThread_45 - putMessage not in lock eclipse time(ms)=559, bodyLength=425 2019-01-25 12:24:07 WARN SendMessageThread_26 - putMessage not in lock eclipse time(ms)=692, bodyLength=115 2019-01-25 12:24:07 WARN SendMessageThread_22 - putMessage not in lock eclipse time(ms)=602, bodyLength=446

Jan 25 sar -B 09时00分01秒 pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff 09时10分01秒 53.90 2150.23 603.07 0.05 6046.29 108.76 0.00 104.54 96.12 09时20分01秒 68.76 2037.86 597.67 0.06 6079.41 135.21 0.00 127.00 93.93 09时30分01秒 41.85 1953.13 593.96 0.04 5986.48 41.96 0.00 38.02 90.60 09时40分01秒 218.06 1910.26 591.13 0.14 6039.40 176.76 0.00 162.66 92.02 09时50分01秒 259.03 2130.62 597.69 0.14 6020.30 174.67 0.00 153.54 87.90 10时00分01秒 12.83 1944.46 587.39 0.02 6002.22 75.59 0.00 67.29 89.01 10时10分01秒 334.58 2678.21 651.55 0.17 6597.55 198.01 0.00 184.85 93.36 10时20分01秒 65.39 2727.90 649.05 0.05 6533.37 143.67 0.00 133.15 92.67 10时30分01秒 71.57 2843.67 649.52 0.05 6519.89 148.09 0.00 131.89 89.07 10时40分01秒 598.96 2456.35 618.90 0.27 6377.77 271.93 0.00 238.88 87.85 10时50分01秒 681.50 3493.50 708.71 0.32 7013.90 328.37 0.00 286.70 87.31 11时00分01秒 2133.78 5041.79 1115.00 1.54 10959.66 1752.14 0.00 1504.10 85.84 11时10分01秒 710.04 4035.66 780.01 0.29 11366.08 687.29 0.00 423.10 61.56 11时20分01秒 222.00 1665.62 572.50 0.11 10287.25 158.53 0.00 99.28 62.63 11时30分01秒 189.29 1588.65 571.60 0.10 23648.61 146.99 0.00 95.86 65.22 11时40分01秒 223.05 1519.33 570.70 0.15 6894.66 293.27 0.00 115.52 39.39 11时50分01秒 235.13 1918.83 571.58 0.16 6124.60 459.42 0.00 159.91 34.81 12时00分01秒 199.34 1768.49 588.22 0.15 8396.45 153.84 0.00 72.71 47.26 12时10分01秒 1004.09 1763.49 693.69 0.46 8764.61 930.06 4.40 377.07 40.35 12时20分01秒 961.36 1533.94 580.84 0.43 8560.14 780.52 0.00 321.29 41.16 12时30分01秒 1014.18 1687.05 576.44 0.49 8590.14 457.40 1.61 270.16 58.86 12时40分01秒 993.98 1601.02 571.71 0.46 8627.73 474.87 0.00 289.02 60.86 12时50分01秒 636.37 1550.81 565.83 0.34 8576.74 243.91 0.00 239.92 98.37 13时00分01秒 256.55 1513.39 564.31 0.21 8460.28 154.36 0.00 149.51 96.85 13时10分01秒 523.77 3249.68 716.50 0.28 11102.90 285.32 0.00 278.41 97.58 13时20分01秒 750.04 3301.08 726.68 0.35 11199.65 308.59 0.00 303.10 98.22 13时30分01秒 744.75 3240.18 678.40 0.36 10701.32 418.28 0.00 299.91 71.70 13时40分01秒 969.07 2451.22 629.35 0.44 10050.09 487.96 0.00 357.73 73.31 13时50分01秒 592.18 2429.16 629.88 0.31 9914.20 442.84 0.00 290.90 65.69 14时00分01秒 448.55 1771.07 585.40 0.27 8824.74 139.92 0.00 118.98 85.03 14时10分01秒 657.93 1832.27 634.43 0.39 8736.66 295.38 0.00 248.61 84.17 平均时间: 907.71 1684.08 619.83 0.41 6832.66 409.30 0.10 316.49 77.31

提问110.png

rocketmq version is 4.3.0

原提问者GitHub用户sN0wpeak

展开
收起
芬奇福贵 2023-05-26 16:10:42 261 0
1 条回答
写回答
取消 提交回答
  • 正如您在本期文章中所描述的,您在一个8核cpu服务器中部署了RocketMQ代理,但正如日志所示,配置中有超过48个sendMessageThread,并且RocketMQ使用spinlock作为默认值,因此您可以尝试将sendMessageThreadnum调整为8,以防止更多线程被阻止以获得spinlock,这也会降低您的cpu实用性。

    请更多地关注您的内存使用情况,尤其是缓存使用情况,请为RocketMQ保留相对更多的可用内存。

    原回答者GitHub用户duhenglucky

    2023-05-26 18:04:17
    赞同 展开评论 打赏

阿里云拥有国内全面的云原生产品技术以及大规模的云原生应用实践,通过全面容器化、核心技术互联网化、应用 Serverless 化三大范式,助力制造业企业高效上云,实现系统稳定、应用敏捷智能。拥抱云原生,让创新无处不在。

相关电子书

更多
低代码开发师(初级)实战教程 立即下载
冬季实战营第三期:MySQL数据库进阶实战 立即下载
阿里巴巴DevOps 最佳实践手册 立即下载