大神,好!
在使用Otter的时候,发现channel挂起,看日志,发现类似下面错误日志。 在这期间主库添加了字段,数据量快2000万。
看了一些源码,感觉如果开启跳过ddl异常开关,如果由于网络抖动或者其他原因,会不会真的没有执行ddl?如果关闭跳过ddl异常开关,又一直挂起,manager管理页面操作解挂,没有解决问题。
版本,4.2.17
请问如何处理?
2019-06-18 17:25:35.166 [destination = 192.168.56.21 , address = /192.168.56.10:3306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - ---> find start position successfully, EntryPosition[included=false,journalName= mysql-bin.000005,position=247775225,serverId=2,gtid=,timestamp=1560849905000] cost : 498ms , the next step is binlog dump 2019-06-18 17:25:43.815 [destination = 192.168.56.21 , address = /192.168.56.10:3306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - ERROR ## parse this event has an error , last position : [EntryPosition[included =false,journalName=mysql-bin.000005,position=252882764,serverId=2,gtid=,timestamp=1560849914000]] com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: column size is not match for table:xxl.student,6 vs 5 2019-06-18 17:25:43.815 [destination = 192.168.56.21 , address = /192.168.56.10:3306 , EventParser] ERROR c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - dump address 192.168.56.10/192.168.56.10:3306 has an error, retrying. caused by com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: column size is not match for table:xxl.student,6 vs 5 2019-06-18 17:25:43.816 [destination = 192.168.56.21 , address = /192.168.56.10:3306 , EventParser] WARN c.a.o.s.a.i.setl.zookeeper.termin.WarningTerminProcess - nid:1[1:canal:192.168.56.21:com.alibaba.otter.canal.parse.exception.CanalParse Exception: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: column size is not match for table:xxl.student,6 vs 5 ] 2019-06-18 17:25:44.989 [pipelineId = 1,taskName = LoadWorker] ERROR com.alibaba.otter.node.etl.load.LoadTask - [1] loadWork executor is error! data:EtlEventData[currNid=1,nextNid=1,desc=[MemoryPipeKey[identity=Identity[channelId=1,pipelineId=1,processId=58],time=1560849943808,dataType=DB_BATCH]],processId=58,startTime=1560849943801,endTime=,firstTime=1560849908000,batchId=60,number=1,size=,exts=,pipelineId=1] com.alibaba.otter.node.etl.load.exception.LoadException: java.util.concurrent.ExecutionException: com.alibaba.otter.node.etl.load.exception.LoadException: com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column name 'is_delete' Caused by: java.util.concurrent.ExecutionException: com.alibaba.otter.node.etl.load.exception.LoadException: com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column name 'is_delete' at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_212] at java.util.concurrent.FutureTask.get(FutureTask.java:192) [na:1.8.0_212] at com.alibaba.otter.node.etl.load.loader.db.DataBatchLoader.load(DataBatchLoader.java:107) ~[node.etl-4.2.17.jar:na] at com.alibaba.otter.node.etl.load.loader.OtterLoaderFactory.load(OtterLoaderFactory.java:50) ~[node.etl-4.2.17.jar:na] at com.alibaba.otter.node.etl.load.LoadTask$1.run(LoadTask.java:85) ~[node.etl-4.2.17.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212] Caused by: com.alibaba.otter.node.etl.load.exception.LoadException: com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysq l.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column name 'is_delete' Caused by: com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplic ate column name 'is_delete' Caused by: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column name 'is_delete' at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:98) ~[spring-jdbc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) ~[spring-jdbc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407) ~[spring-jdbc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction.doDdl(DbLoadAction.java:357) ~[node.etl-4.2.17.jar:na]
原提问者GitHub用户hudajin1980
根据您提供的错误日志,看起来是由于主库在添加字段后,数据表的列数和数据包中的列数不匹配,导致了 Canal 无法解析数据包。这可能是由于 Otter 所使用的 Canal 版本不支持自动添加列的功能,或者主库中的添加列操作与 Otter 中的数据表定义不一致导致的。
在这种情况下,您可以尝试执行以下步骤来解决问题:
确认 Otter 中的数据表定义与主库中的数据表定义一致。 如果您有必要添加新的列,请手动将其添加到 Otter 中的数据表定义中。 确认 Otter 所使用的 Canal 版本是否支持自动添加列的功能。如果不支持,请手动添加新的列。 如果您仍然遇到问题,请尝试在 Otter 配置文件中关闭跳过 ddl 异常开关,以确保 Otter 可以正确执行主库中的 ddl 操作。 如果以上步骤都不能解决问题,请考虑升级 Otter 或 Canal 版本,或者联系 Otter 官方支持团队以获取更多帮助。
Duplicate column name 'is_delete',应该是重复的ddl同步,可以高级设置里关闭DDL复制
原回答者GitHub用户 agapple
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。