开发者社区> 问答> 正文

Canal-adapter-1.1.3 mysql 同步数据到 elasticsearch 报错

2019-04-11 16:49:30.532 [pool-2-thread-1] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - java.lang.RuntimeException: java.lang.RuntimeException: TransportSerializationException[Failed to deserialize response from handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler]]; nested: IllegalArgumentException[Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][ml]]; java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: TransportSerializationException[Failed to deserialize response from handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler]]; nested: IllegalArgumentException[Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][ml]]; at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:113) at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:61) at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:176) at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:155) at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.batchSync(AbstractCanalAdapterWorker.java:198) at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$1(AbstractCanalAdapterWorker.java:62) at java.util.ArrayList.forEach(ArrayList.java:1249) at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$2(AbstractCanalAdapterWorker.java:58) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: TransportSerializationException[Failed to deserialize response from handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler]]; nested: IllegalArgumentException[Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][ml]]; at com.alibaba.otter.canal.client.adapter.support.Util.sqlRS(Util.java:47) at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.mainTableInsert(ESSyncService.java:468) at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.insert(ESSyncService.java:140) at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:96) ... 11 common frames omitted Caused by: java.lang.RuntimeException: TransportSerializationException[Failed to deserialize response from handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler]]; nested: IllegalArgumentException[Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][ml]]; at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.lambda$mainTableInsert$3(ESSyncService.java:485) at com.alibaba.otter.canal.client.adapter.support.Util.sqlRS(Util.java:43) ... 14 common frames omitted Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response from handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler] at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:1441) at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1400) at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:64) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ... 1 common frames omitted Caused by: java.lang.IllegalArgumentException: Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][ml] at org.elasticsearch.common.io.stream.NamedWriteableRegistry.getReader(NamedWriteableRegistry.java:112) at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:45) at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:39) at org.elasticsearch.cluster.metadata.MetaData.readFrom(MetaData.java:775) at org.elasticsearch.cluster.ClusterState.readFrom(ClusterState.java:668) at org.elasticsearch.action.admin.cluster.state.ClusterStateResponse.readFrom(ClusterStateResponse.java:81) at org.elasticsearch.transport.TransportResponseHandler.read(TransportResponseHandler.java:47) at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1085) at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:1437) ... 27 common frames omitted 2019-04-11 16:49:30.534 [Thread-4] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - Outer adapter sync failed! Error sync but ACK! 2019-04-11 16:49:30.534 [elasticsearch[client][transport_client_boss][T#3]] WARN org.elasticsearch.transport.netty4.Netty4Transport - exception caught on transport layer [NettyTcpChannel{localAddress=/192.168.100.207:59176, remoteAddress=/192.168.100.204:9300}], closing connection java.lang.IllegalStateException: Message not fully read (response) for requestId [13], handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.action.ActionListenerResponseHandler@58cac2ff], error [false]; resetting at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1407) at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:64) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at java.lang.Thread.run(Thread.java:748)

elasticsearch server版本号是6.2.4 Canal server版本号1.1.3 Canal-adapter版本号1.1.3

mysql数据变化时,canal-adapter得到了数据, 但是向elasticsearch同步数据时,抛异常了

原提问者GitHub用户zhuxiaokang0223

展开
收起
古拉古拉 2023-05-08 12:35:38 74 0
2 条回答
写回答
取消 提交回答
  • 应该是 es 配置里面,设置的 _id : id 造成的吧,你的sql语句里面最后select出来的实际上只有 _id字段

    原回答者GitHub用户simonlei

    2023-05-09 17:38:25
    赞同 展开评论 打赏
  • 值得去的地方都没有捷径

    根据您提供的信息,Canal-adapter-1.1.3 在将 MySQL 数据同步到 Elasticsearch 时,出现了 TransportSerializationException 异常。这个异常通常是由 Elasticsearch 版本不兼容引起的。具体而言,Elasticsearch 6.x 版本中引入了 NamedWriteable 接口,用于序列化和反序列化 Elasticsearch 中的对象。如果使用的 Elasticsearch 版本低于 6.x,就会出现 TransportSerializationException 异常。

    解决这个问题的方法是,将 Elasticsearch 升级到 6.x 或以上版本。同时,您需要确保 Canal-adapter-1.1.3 和 Elasticsearch 版本兼容。如果您使用的是 Canal-adapter-1.1.3,建议将 Elasticsearch 升级到 6.x 版本,并使用 Elasticsearch 6.x 版本的 Java API 进行开发。如果您使用的是其他版本的 Canal-adapter,建议查看对应版本的文档,以确定与 Elasticsearch 版本的兼容性。

    另外,如果您已经将 Elasticsearch 升级到 6.x 或以上版本,但仍然出现 TransportSerializationException 异常,可能是由于 Elasticsearch 集群中存在不同版本的节点,导致序列化和反序列化出现问题。这种情况下,您需要检查 Elasticsearch 集群中的节点版本是否一致,并确保所有节点都升级到了 6.x 或以上版本。

    2023-05-08 12:49:33
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
阿里云Elasticsearch体系架构与特性解析 立即下载
开源与云:Elasticsearch应用剖析 立即下载
《Elasticsearch全观测解决方案》 立即下载

相关镜像