问题情况
从同步变慢直到无法获取数据且无法恢复
详细描述
有一台RDS部署在德国, 国内需要同步这台RDS的数据,所以在国内部署了一台Canal-Server获取该RDS的binlog. 刚启动的那一段时间内还好一切正常,之后同步就开始逐渐变慢再之后(不会超过1天)就彻底无法获取数据了。 初步判断是因为国内外网络连接引起的问题,从Canal-Server的log来看经常会有timeout的异常,我也找过类似的issue看过很多给出的解决方案都是客户端做好重试,然后我也尝试做了重试,客户端捕获CanalClientException,出现异常后断开连接再重连, 但依然无法解决我的问题。
环境
Canal 1.1.0 MySQL 5.6
异常堆栈
Canal-Server 2019-09-25 14:53:19.582 [destination = sync , address = /* , EventParser] ERROR c.a.o.canal.parse.inbound.mysql.dbsync.DirectLogFetcher - Socket timeout expired, closing connection java.net.SocketTimeoutException: Timeout occurred, failed to read 8623 bytes in 25000 milliseconds. at com.alibaba.otter.canal.parse.driver.mysql.socket.BioSocketChannel.read(BioSocketChannel.java:123) ~[canal.parse.driver-1.1.0.jar:na] at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch0(DirectLogFetcher.java:174) ~[canal.parse-1.1.0.jar:na] at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:85) ~[canal.parse-1.1.0.jar:na] at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:153) [canal.parse-1.1.0.jar:na] at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:262) [canal.parse-1.1.0.jar:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] 2019-09-25 14:53:19.584 [destination = sync , address = /* , EventParser] ERROR c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - dump address /* has an error, retrying. caused by java.net.SocketTimeoutException: Timeout occurred, failed to read 8623 bytes in 25000 milliseconds. at com.alibaba.otter.canal.parse.driver.mysql.socket.BioSocketChannel.read(BioSocketChannel.java:123) ~[canal.parse.driver-1.1.0.jar:na] at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch0(DirectLogFetcher.java:174) ~[canal.parse-1.1.0.jar:na] at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:85) ~[canal.parse-1.1.0.jar:na] at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:153) ~[canal.parse-1.1.0.jar:na] at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:262) ~[canal.parse-1.1.0.jar:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] 2019-09-25 14:53:19.590 [destination = sync , address = /* , EventParser] ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandler - destination:sync[java.net.SocketTimeoutException: Timeout occurred, failed to read 8623 bytes in 25000 milliseconds. at com.alibaba.otter.canal.parse.driver.mysql.socket.BioSocketChannel.read(BioSocketChannel.java:123) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch0(DirectLogFetcher.java:174) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:85) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:153) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:262) at java.lang.Thread.run(Thread.java:748)
Canal-Client something goes wrong when acking data from server:null batchId:305 is not exist , please check
原提问者GitHub用户wbsch52
1、建议先升级为1.1.4版本
2、这个超时机制主要是源库一直没有binlog,可以考虑开启心跳SQL来定期产生binlog
原回答者GitHub用户agapple
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。