开发者社区 > 大数据与机器学习 > 实时计算 Flink > 正文

"Flink CDC中这个问题怎么解决?

"Flink CDC中这个问题怎么解决?Failed to deserialize data of EventHeaderV4这个问题。
2023-07-15 05:40:09,119 ERROR io.debezium.pipeline.ErrorHandler [] - Producer failure
io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1689370731000, eventType=ROWS_QUERY, serverId=683728822, headerLength=19, dataLength=311, nextPosition=206875527, flags=128}
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1489) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(MySqlStreamingChangeEventSource.java:1545) [blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1079) [blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631) [blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932) [blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1689370731000, eventType=ROWS_QUERY, serverId=683728822, headerLength=19, dataLength=311, nextPosition=206875527, flags=128}
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:341) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:244) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:260) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1051) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
... 3 more
Caused by: java.io.EOFException: Failed to read remaining 72 of 306 bytes from position 3501350. Block length: 72. Initial block length: 307.
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.fill(ByteArrayInputStream.java:115) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:105) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readString(ByteArrayInputStream.java:78) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.event.deserialization.RowsQueryEventDataDeserializer.deserialize(RowsQueryEventDataDeserializer.java:31) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.event.deserialization.RowsQueryEventDataDeserializer.deserialize(RowsQueryEventDataDeserializer.java:25) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:335) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:244) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:260) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1051) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]"

展开
收起
小易01 2023-07-24 21:52:37 547 0
4 条回答
写回答
取消 提交回答
  • 作者这个问题有解决吗? 遇到同样问题 求救。。。

    2024-01-24 10:44:59
    赞同 展开评论 打赏
  • 北京阿里云ACE会长

    在 Flink CDC 中,"Failed to deserialize data" 错误通常表示在将数据反序列化为对象时出现了问题。可能原因包括:
    数据格式不正确:数据源发送的数据格式可能不符合期望的格式,或者与 Flink 应用程序中定义的数据类型不匹配,导致反序列化失败。
    数据类型不匹配:Flink 应用程序中定义的数据类型与实际数据类型不匹配,导致反序列化失败。
    序列化/反序列化库版本不匹配:Flink 应用程序使用的序列化/反序列化库版本与数据源发送的数据使用的版本不匹配,导致反序列化失败。

    2023-07-29 18:35:26
    赞同 展开评论 打赏
  • 意中人就是我呀!

    把binlog清了再启动任务看下。sql可以的话,把你的ds代码打个包到集群运行,直print连接器,看看有么有报错,不上生产。 sql要是没问题,大包包ds的代码上集群运行看看有没有报错。此回答整理至钉群“Flink CDC 社区”

    2023-07-26 12:01:22
    赞同 展开评论 打赏
  • 存在即是合理

    这个错误提示表明 Flink CDC 在反序列化 EventHeaderV4 数据时失败了。可能是因为数据格式不正确或者缺少必要的字段。可以尝试检查MySQL binlog 格式是否正确,并确保所有必要的字段都存在。另外,也可以查看 Flink CDC 的文档,了解如何正确配置 MySQL CDC。

    2023-07-25 14:27:04
    赞同 展开评论 打赏

实时计算Flink版是阿里云提供的全托管Serverless Flink云服务,基于 Apache Flink 构建的企业级、高性能实时大数据处理系统。提供全托管版 Flink 集群和引擎,提高作业开发运维效率。

相关产品

  • 实时计算 Flink版
  • 相关电子书

    更多
    Flink CDC Meetup PPT - 龚中强 立即下载
    Flink CDC Meetup PPT - 王赫 立即下载
    Flink CDC Meetup PPT - 覃立辉 立即下载