开发者社区 > 大数据与机器学习 > 实时计算 Flink > 正文

flinkcdc我们在生产环境遇到一个非常奇怪的问题要怎么解决?

我们在生产环境遇到一个非常奇怪的问题,CDC在监听到一个TABLE_MAP事件后,在context里找不到表信息而出现了... whose schema isn't known to this connector异常,怀疑可能和这里的TODO有关系导致schema里没有这个表。6c4f15e25c6663c6fe191423005e4a87.png2024-03-01 00:00:07,215 ERROR io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Encountered change event 'Event{header=EventHeaderV4{timestamp=1709222407000, eventType=TABLE_MAP, serverId=34153158, headerLength=19, dataLength=62, nextPosition=459864703, flags=0}, data=TableMapEventData{tableId=4322148, database='dj_log', table='t_map_crash_log_202403', columnTypes=3, 8, 8, 3, 15, 15, 15, 18, columnMetadata=0, 0, 0, 0, 256, 1020, 2048, 0, columnNullability={}, eventMetadata=null}}' at offset {transaction_id=null, ts_sec=1709222392, file=binlog.010032, pos=459863947, gtids=0f6fc04c-6a08-11ed-bcb2-f29b611b6193:1-14394004114, server_id=34153158, event=2} for table dj_log.t_map_crash_log_202403 whose schema isn't known to this connector. One possible cause is an incomplete database history topic. Take a new snapshot in this case.
Use the mysqlbinlog tool to view the problematic event: mysqlbinlog --start-position=459864622 --stop-position=459864703 --verbose binlog.010032

2024-03-01 00:00:07,215 ERROR io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Error during binlog processing. Last offset stored = null, binlog reader near position = binlog.010032/459864622

2024-03-01 00:00:07,215 WARN com.ververica.cdc.connectors.mysql.debezium.task.context.MySqlErrorHandler - Schema for table dj_log.t_map_crash_log_202403 is null

2024-03-01 00:00:07,215 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shutdown will be ignored.然后这个问题在2.3之后修复了?这个有相关的pr或者什么资料可以参考下嘛?

展开
收起
小小鹿鹿鹿 2024-03-25 19:45:31 24 0
1 条回答
写回答
取消 提交回答
  • 换2.3以上。批量操作数据,even t读取过长。此回答来自钉群Flink CDC 社区。

    2024-03-25 22:49:15
    赞同 展开评论 打赏

实时计算Flink版是阿里云提供的全托管Serverless Flink云服务,基于 Apache Flink 构建的企业级、高性能实时大数据处理系统。提供全托管版 Flink 集群和引擎,提高作业开发运维效率。

相关产品

  • 实时计算 Flink版
  • 相关电子书

    更多
    Flink CDC Meetup PPT - 龚中强 立即下载
    Flink CDC Meetup PPT - 王赫 立即下载
    Flink CDC Meetup PPT - 覃立辉 立即下载