客户端发送ack请求异常，重试之后获取节点一直报错

环境信息

canal version 1.1.3 mysql version 5.7.22

问题描述

发现利用canal-adapter模块 canal同步数据到es时候，发现客户端往服务端发送ack信息的时候，服务端发生异常，随后客户端重试发送获取最新节点信息的时候，一直报节点不存在错误。

2019-03-27 23:14:48.299 WARN 24460 --- [ Thread-32] c.a.o.c.c.impl.ClusterCanalConnector : something goes wrong when acking data from server:null

com.alibaba.otter.canal.protocol.exception.CanalClientException: java.io.IOException: Broken pipe at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.ack(SimpleCanalConnector.java:342) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.ClusterCanalConnector.ack(ClusterCanalConnector.java:252) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterWorker.process(CanalAdapterWorker.java:128) [client-adapter.launcher-1.1.3-aifocus-RELEASE.jar:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_131] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_131] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_131] at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_131] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_131] at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) ~[na:1.8.0_131] at java.nio.channels.Channels.writeFully(Channels.java:98) ~[na:1.8.0_131] at java.nio.channels.Channels.access$000(Channels.java:61) ~[na:1.8.0_131] at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.8.0_131] at java.nio.channels.Channels$WritableByteChannelImpl.write(Channels.java:458) ~[na:1.8.0_131] at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.writeWithHeader(SimpleCanalConnector.java:385) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.writeWithHeader(SimpleCanalConnector.java:372) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.ack(SimpleCanalConnector.java:336) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] ... 3 common frames omitted

2019-03-27 23:14:53.348 INFO 24460 --- [ Thread-32] c.a.o.c.c.impl.ClusterCanalConnector : restart the connector for next round retry. 2019-03-27 23:14:53.387 WARN 24460 --- [ Thread-32] c.a.o.c.c.impl.ClusterCanalConnector : something goes wrong when getWithoutAck data from server:null

com.alibaba.otter.canal.protocol.exception.CanalClientException: deserializer failed at com.alibaba.otter.canal.client.CanalMessageDeserializer.deserializer(CanalMessageDeserializer.java:54) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.receiveMessages(SimpleCanalConnector.java:322) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.getWithoutAck(SimpleCanalConnector.java:314) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.getWithoutAck(SimpleCanalConnector.java:286) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.ClusterCanalConnector.getWithoutAck(ClusterCanalConnector.java:183) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterWorker.process(CanalAdapterWorker.java:105) [client-adapter.launcher-1.1.3-aifocus-RELEASE.jar:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] Caused by: com.alibaba.otter.canal.protocol.exception.CanalClientException: something goes wrong with reason: something goes wrong with channel:[id: 0x45e361b4, /172.16.1.119:54662 => /172.16.1.119:11111], exception=com.alibaba.otter.canal.server.exception.CanalServerException: ack error , clientId:1001 batchId:318200 is not exist , please check

at com.alibaba.otter.canal.client.CanalMessageDeserializer.deserializer(CanalMessageDeserializer.java:46) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] ... 6 common frames omitted

2019-03-27 23:14:58.436 INFO 24460 --- [ Thread-32] c.a.o.c.c.impl.ClusterCanalConnector : restart the connector for next round retry. 2019-03-28 01:14:04.710 WARN 24460 --- [ Thread-32] c.a.o.c.c.impl.ClusterCanalConnector : something goes wrong when acking data from server:null

com.alibaba.otter.canal.protocol.exception.CanalClientException: java.io.IOException: Broken pipe at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.ack(SimpleCanalConnector.java:342) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.ClusterCanalConnector.ack(ClusterCanalConnector.java:252) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterWorker.process(CanalAdapterWorker.java:128) [client-adapter.launcher-1.1.3-aifocus-RELEASE.jar:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_131] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_131] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_131] at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_131] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_131] at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) ~[na:1.8.0_131] at java.nio.channels.Channels.writeFully(Channels.java:98) ~[na:1.8.0_131] at java.nio.channels.Channels.access$000(Channels.java:61) ~[na:1.8.0_131] at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.8.0_131] at java.nio.channels.Channels$WritableByteChannelImpl.write(Channels.java:458) ~[na:1.8.0_131] at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.writeWithHeader(SimpleCanalConnector.java:385) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.writeWithHeader(SimpleCanalConnector.java:372) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.ack(SimpleCanalConnector.java:336) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] ... 3 common frames omitted

2019-03-28 01:14:09.757 INFO 24460 --- [ Thread-32] c.a.o.c.c.impl.ClusterCanalConnector : restart the connector for next round retry. 2019-03-28 01:14:09.796 WARN 24460 --- [ Thread-32] c.a.o.c.c.impl.ClusterCanalConnector : something goes wrong when getWithoutAck data from server:null

com.alibaba.otter.canal.protocol.exception.CanalClientException: deserializer failed at com.alibaba.otter.canal.client.CanalMessageDeserializer.deserializer(CanalMessageDeserializer.java:54) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.receiveMessages(SimpleCanalConnector.java:322) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.getWithoutAck(SimpleCanalConnector.java:314) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.getWithoutAck(SimpleCanalConnector.java:286) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.client.impl.ClusterCanalConnector.getWithoutAck(ClusterCanalConnector.java:183) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterWorker.process(CanalAdapterWorker.java:105) [client-adapter.launcher-1.1.3-aifocus-RELEASE.jar:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] Caused by: com.alibaba.otter.canal.protocol.exception.CanalClientException: something goes wrong with reason: something goes wrong with channel:[id: 0x6afd2e2e, /172.16.1.119:55880 => /172.16.1.119:11111], exception=com.alibaba.otter.canal.server.exception.CanalServerException: ack error , clientId:1001 batchId:318202 is not exist , please check

at com.alibaba.otter.canal.client.CanalMessageDeserializer.deserializer(CanalMessageDeserializer.java:46) ~[canal.client-1.1.3-aifocus-RELEASE.jar:na] ... 6 common frames omitted

2019-03-28 01:14:14.844 INFO 24460 --- [ Thread-32] c.a.o.c.c.impl.ClusterCanalConnector : restart the connector for next round retry.

原提问者GitHub用户chenjgtea