Flink CDC中使用TimeStamp方式的mysql datasource会包内存溢出,时间开始时间就是当开开始10分钟。大家遇到过没
2023-07-28 11:18:41,709 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Keepalive thread is running
2023-07-28 11:18:43,677 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 1: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:18:48,611 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 2: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:18:53,601 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 3: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:18:58,608 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 4: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:19:03,601 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 5: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:19:08,592 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 6: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:19:13,608 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 7: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:19:18,604 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 8: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
2023-07-28 11:19:21,100 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Stopped reading binlog after 0 events, no new offset was recorded
2023-07-28 11:19:21,101 ERROR org.apache.flink.runtime.util.ClusterUncaughtExceptionHandler [] - WARNING: Thread 'blc-192.168.200.120:3306' produced an uncaught exception. If you want to fail on uncaught exceptions, then configure cluster.uncaught-exception-handling accordingly
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236) ~[?:1.8.0_311]
at com.github.shyiko.mysql.binlog.BinaryLogClient.readPacketSplitInChunks(BinaryLogClient.java:1099) ~[drs-server.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1052) ~[drs-server.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631) ~[drs-server.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932) ~[drs-server.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_311]
2023-07-28 11:19:23,598 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 9: {ts_sec=1690509600, kind=TIMESTAMP, row=0, event=0}
Flink CDC中使用的Debezium MySQL Connectors版本有关,因为在Debezium MySQL Connectors 1.5.0之前的版本中,使用Timestamp类型的binlog event时可能会出现内存泄漏的问题。这个问题已经在Debezium MySQL Connectors 1.5.0中得到了修复。
https://help.aliyun.com/document_detail/2382880.html?spm=a2c4g.11186623.0.i31
因此,如果你使用的是早于Debezium MySQL Connectors 1.5.0的版本,建议升级到最新版本并尝试再次运行。如果问题仍然存在,请检查系统的内存使用情况,尝试调整JVM的内存配置或者使用更高配置的机器来运行。
另外,需要注意的是,在Flink CDC中使用Timestamp类型的binlog event时,需要保证系统的时间同步和时区设置正确,否则可能会导致binlog event时间戳和实际时间不一致,从而导致数据同步错误。
https://help.aliyun.com/document_detail/44213.html?spm=a2c4g.2382880.0.i8
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。
实时计算Flink版是阿里云提供的全托管Serverless Flink云服务,基于 Apache Flink 构建的企业级、高性能实时大数据处理系统。提供全托管版 Flink 集群和引擎,提高作业开发运维效率。