开发者社区 > 大数据与机器学习 > 实时计算 Flink > 正文

Flink CDC中使用TimeStamp方式的mysql datasource会包内存溢出,时间?

Flink CDC中使用TimeStamp方式的mysql datasource会包内存溢出,时间开始时间就是当开开始10分钟。大家遇到过没
2023-07-28 11:18:41,709 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Keepalive thread is running
2023-07-28 11:18:43,677 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 1: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:18:48,611 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 2: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:18:53,601 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 3: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:18:58,608 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 4: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:19:03,601 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 5: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:19:08,592 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 6: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:19:13,608 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 7: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:19:18,604 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 8: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:19:21,100 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Stopped reading binlog after 0 events, no new offset was recorded
2023-07-28 11:19:21,101 ERROR org.apache.flink.runtime.util.ClusterUncaughtExceptionHandler [] - WARNING: Thread 'blc-192.168.200.120:3306' produced an uncaught exception. If you want to fail on uncaught exceptions, then configure cluster.uncaught-exception-handling accordingly
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236) ~[?:1.8.0_311]
at com.github.shyiko.mysql.binlog.BinaryLogClient.readPacketSplitInChunks(BinaryLogClient.java:1099) ~[drs-server.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1052) ~[drs-server.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631) ~[drs-server.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932) ~[drs-server.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_311]
2023-07-28 11:19:23,598 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 9: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}

展开
收起
真的很搞笑 2023-08-01 15:16:22 136 0
1 条回答
写回答
取消 提交回答
  • 北京阿里云ACE会长

    Flink CDC中使用的Debezium MySQL Connectors版本有关,因为在Debezium MySQL Connectors 1.5.0之前的版本中,使用Timestamp类型的binlog event时可能会出现内存泄漏的问题。这个问题已经在Debezium MySQL Connectors 1.5.0中得到了修复。

    https://help.aliyun.com/document_detail/2382880.html?spm=a2c4g.11186623.0.i31

    image.png

    因此,如果你使用的是早于Debezium MySQL Connectors 1.5.0的版本,建议升级到最新版本并尝试再次运行。如果问题仍然存在,请检查系统的内存使用情况,尝试调整JVM的内存配置或者使用更高配置的机器来运行。

    另外,需要注意的是,在Flink CDC中使用Timestamp类型的binlog event时,需要保证系统的时间同步和时区设置正确,否则可能会导致binlog event时间戳和实际时间不一致,从而导致数据同步错误。

    https://help.aliyun.com/document_detail/44213.html?spm=a2c4g.2382880.0.i8
    image.png

    2023-08-01 17:43:21
    赞同 展开评论 打赏

实时计算Flink版是阿里云提供的全托管Serverless Flink云服务,基于 Apache Flink 构建的企业级、高性能实时大数据处理系统。提供全托管版 Flink 集群和引擎,提高作业开发运维效率。

相关产品

  • 实时计算 Flink版
  • 相关电子书

    更多
    搭建电商项目架构连接MySQL 立即下载
    搭建4层电商项目架构,实战连接MySQL 立即下载
    PolarDB MySQL引擎重磅功能及产品能力盛大发布 立即下载

    相关镜像