当时的现象:
客户端报错日志:
^T0.000000000000000000^X^R�^A^B�^A��ި% to topic FINISHO_T_CTU fail: org.apache.rocketmq.remoting.exception.RemotingTimeoutException: wait response on the channel <10.10.42.69:10911> timeout, 1000(ms) at org.apache.rocketmq.remoting.netty.NettyRemotingAbstract.invokeSyncImpl(NettyRemotingAbstract.java:386) at org.apache.rocketmq.remoting.netty.NettyRemotingClient.invokeSync(NettyRemotingClient.java:369) at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessageSync(MQClientAPIImpl.java:351) at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:335) at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:298) at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendKernelImpl(DefaultMQProducerImpl.java:696) at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendSelectImpl(DefaultMQProducerImpl.java:914) at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:891) at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:886) at org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:389) at io.ebonex.base.mq.MessageProducer.doRealSendOrderly(MessageProducer.java:255) at io.ebonex.base.mq.MessageProducer.doRealSendOrderly(MessageProducer.java:200) at io.ebonex.base.mq.MessageProducer.sendOrderly(MessageProducer.java:155) at io.ebonex.ex.match.queue.TradeProducer$4.run(TradeProducer.java:689) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829)
服务端的报错日志
closeChannel: close the connection to remote address
很多netty的 关闭日志:
当时的日志截图
问题追踪:
客户端使用的版本:4.2.0
服务端端使用的版本:4.8,
排查过程:
默认是开放 vip 端口
分别为(默认配置):
- 9876:nameserver 监听端口
- 10911: broker 监听端口
- 10909:broker vip 监听端口
- 10912:broker HA 端口,用于主从同步
为什么 broker 将会启动两个端口?
rocketmq broker 虽然启动了两个端口,但是从 rocketmq broker 的源码可以发现这两个端口启动之后起到作用是一样的。
那为什么开启两个监听端口那?我想很多同学应该也有这个疑惑,这里给出一个开发者解释答案。
https://github.com/apache/rocketmq/issues/1510
普通的端口将会承载所有消息网络请求,如果此时请求非常繁忙,broker 端的所有 I/O 线程可能都在执行请求,这就会导致后续网络请求进入队列,从而导致消息请求执行缓慢。