问题一:Flink CDC能否实时监控HDFS上的文件变化?
Flink CDC能否实时监控HDFS上的文件变化?例如,能否快速捕获到HDFS某个文件中某条数据的变动?
参考答案:
这种flume可以采集吧。
关于本问题的更多回答可点击进行查看:
https://developer.aliyun.com/ask/596402?spm=a2c6h.12873639.article-detail.102.50e24378TRW91E
问题二:Flink CDC里报这个错是doris内存不够?
Flink CDC里报这个错是doris内存不够?
参考答案:
是内存不够。
关于本问题的更多回答可点击进行查看:
https://developer.aliyun.com/ask/596392?spm=a2c6h.12873639.article-detail.103.50e24378TRW91E
问题三:Flink CDC里MySQL CDC Pipeline 整库同步时 从指定的时间戳开始读取 吗?
Flink CDC里MySQL CDC Pipeline 整库同步时 从指定的时间戳开始读取 binlog 事件,然后 在指定的时间戳 结束任务 吗?目前看文档只能配置开始吗?能否指定结束时间呢 或者有其他方式实现么?
参考答案:
不建议用现在的pipeline,首先就是没法用组件调度,其次配的参数也不外暴露,你如过同步到doris,你可以直接用doris官方的flink连接器配合cdc一起使用。
关于本问题的更多回答可点击进行查看:
https://developer.aliyun.com/ask/596389?spm=a2c6h.12873639.article-detail.104.50e24378TRW91E
问题四:Flink CDC里在生产跑 MySQL 向 Doris 同步数据,可以直接用 这个吗?
Flink CDC里在生产跑 MySQL 向 Doris 同步数据,可以直接用 QuickStart 里介绍的一样,通过 yaml 来编写 task 配置信息,cdc 脚本启动 flink job 就可以了么,区别只是生产要一个稳定的 flink 集群?
因为我看 flink-cdc 的release 包里,其实就是依赖 cdc 的包和 pipline 的包,pipline 的话是对 connector 进一步的封装。启动任务的 shell 脚本本质也是向 flink 集群提交了作业的吧?
参考答案:
其实就是2.x的更一步封装,把sink端都实现了,形成一个完整的flink job(pipeline)
关于本问题的更多回答可点击进行查看:
https://developer.aliyun.com/ask/596385?spm=a2c6h.12873639.article-detail.105.50e24378TRW91E
问题五:Flink CDC里使用3.0 测试mysql到starrocks启动报错怎么办?
Flink CDC里使用3.0 测试mysql到starrocks启动报错怎么办?
Caused by: org.apache.flink.util.FlinkRuntimeException: com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.openJdbcConnection(DebeziumUtils.java:71)
at com.ververica.cdc.connectors.mysql.MySqlValidator.createJdbcConnection(MySqlValidator.java:83)
at com.ververica.cdc.connectors.mysql.MySqlValidator.validate(MySqlValidator.java:68)
at com.ververica.cdc.connectors.mysql.source.MySqlSource.createEnumerator(MySqlSource.java:197)
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.start(SourceCoordinator.java:221)
... 33 more
Caused by: com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:596)
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:582)
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:115)
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:81)
at com.ververica.cdc.connectors.mysql.source.connection.PooledDataSourceFactory.createPooledDataSource(PooledDataSourceFactory.java:61)
at com.ververica.cdc.connectors.mysql.source.connection.JdbcConnectionPools.getOrCreateConnectionPool(JdbcConnectionPools.java:49)
at com.ververica.cdc.connectors.mysql.source.connection.JdbcConnectionFactory.connect(JdbcConnectionFactory.java:54)
at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:888)
at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:883)
at io.debezium.jdbc.JdbcConnection.connect(JdbcConnection.java:411)
at com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.openJdbcConnection(DebeziumUtils.java:68)
... 37 more
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:829)
at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:449)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:242)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198)
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364)
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476)
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561)
... 46 more
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failureCaused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
Caused by: java.io.EOFException: SSL peer shut down incorrectly
后面还有两个报错
参考答案:
降低下connect.timeout 参数,默认是30s,估计数据库的更短,可以参考mysql的wait_timeout指标,低于这个就行。应该就是这个ssl的问题,你将源端的ssl关闭,或者增加连接参数。 The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.这个错误一般就是连接超时导致的0.0。
关于本问题的更多回答可点击进行查看:
https://developer.aliyun.com/ask/596380?spm=a2c6h.12873639.article-detail.106.50e24378TRW91E