开发者社区> 问答> 正文

otter DDL 出错,日志循环输出column size is not match for ta

大神,好!

在使用Otter的时候,发现channel挂起,看日志,发现类似下面错误日志。 在这期间主库添加了字段,数据量快2000万。

看了一些源码,感觉如果开启跳过ddl异常开关,如果由于网络抖动或者其他原因,会不会真的没有执行ddl?如果关闭跳过ddl异常开关,又一直挂起,manager管理页面操作解挂,没有解决问题。

版本,4.2.17

请问如何处理?

2019-06-18 17:25:35.166 [destination = 192.168.56.21 , address = /192.168.56.10:3306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - ---> find start position successfully, EntryPosition[included=false,journalName= mysql-bin.000005,position=247775225,serverId=2,gtid=,timestamp=1560849905000] cost : 498ms , the next step is binlog dump 2019-06-18 17:25:43.815 [destination = 192.168.56.21 , address = /192.168.56.10:3306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - ERROR ## parse this event has an error , last position : [EntryPosition[included =false,journalName=mysql-bin.000005,position=252882764,serverId=2,gtid=,timestamp=1560849914000]] com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: column size is not match for table:xxl.student,6 vs 5 2019-06-18 17:25:43.815 [destination = 192.168.56.21 , address = /192.168.56.10:3306 , EventParser] ERROR c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - dump address 192.168.56.10/192.168.56.10:3306 has an error, retrying. caused by com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: column size is not match for table:xxl.student,6 vs 5 2019-06-18 17:25:43.816 [destination = 192.168.56.21 , address = /192.168.56.10:3306 , EventParser] WARN c.a.o.s.a.i.setl.zookeeper.termin.WarningTerminProcess - nid:1[1:canal:192.168.56.21:com.alibaba.otter.canal.parse.exception.CanalParse Exception: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: column size is not match for table:xxl.student,6 vs 5 ] 2019-06-18 17:25:44.989 [pipelineId = 1,taskName = LoadWorker] ERROR com.alibaba.otter.node.etl.load.LoadTask - [1] loadWork executor is error! data:EtlEventData[currNid=1,nextNid=1,desc=[MemoryPipeKey[identity=Identity[channelId=1,pipelineId=1,processId=58],time=1560849943808,dataType=DB_BATCH]],processId=58,startTime=1560849943801,endTime=,firstTime=1560849908000,batchId=60,number=1,size=,exts=,pipelineId=1] com.alibaba.otter.node.etl.load.exception.LoadException: java.util.concurrent.ExecutionException: com.alibaba.otter.node.etl.load.exception.LoadException: com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column name 'is_delete' Caused by: java.util.concurrent.ExecutionException: com.alibaba.otter.node.etl.load.exception.LoadException: com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column name 'is_delete' at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_212] at java.util.concurrent.FutureTask.get(FutureTask.java:192) [na:1.8.0_212] at com.alibaba.otter.node.etl.load.loader.db.DataBatchLoader.load(DataBatchLoader.java:107) ~[node.etl-4.2.17.jar:na] at com.alibaba.otter.node.etl.load.loader.OtterLoaderFactory.load(OtterLoaderFactory.java:50) ~[node.etl-4.2.17.jar:na] at com.alibaba.otter.node.etl.load.LoadTask$1.run(LoadTask.java:85) ~[node.etl-4.2.17.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212] Caused by: com.alibaba.otter.node.etl.load.exception.LoadException: com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysq l.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column name 'is_delete' Caused by: com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplic ate column name 'is_delete' Caused by: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column name 'is_delete' at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:98) ~[spring-jdbc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) ~[spring-jdbc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407) ~[spring-jdbc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction.doDdl(DbLoadAction.java:357) ~[node.etl-4.2.17.jar:na]

原提问者GitHub用户hudajin1980

展开
收起
古拉古拉 2023-06-14 21:01:53 244 0
2 条回答
写回答
取消 提交回答
  • 值得去的地方都没有捷径

    根据您提供的错误日志,看起来是由于主库在添加字段后,数据表的列数和数据包中的列数不匹配,导致了 Canal 无法解析数据包。这可能是由于 Otter 所使用的 Canal 版本不支持自动添加列的功能,或者主库中的添加列操作与 Otter 中的数据表定义不一致导致的。

    在这种情况下,您可以尝试执行以下步骤来解决问题:

    确认 Otter 中的数据表定义与主库中的数据表定义一致。 如果您有必要添加新的列,请手动将其添加到 Otter 中的数据表定义中。 确认 Otter 所使用的 Canal 版本是否支持自动添加列的功能。如果不支持,请手动添加新的列。 如果您仍然遇到问题,请尝试在 Otter 配置文件中关闭跳过 ddl 异常开关,以确保 Otter 可以正确执行主库中的 ddl 操作。 如果以上步骤都不能解决问题,请考虑升级 Otter 或 Canal 版本,或者联系 Otter 官方支持团队以获取更多帮助。

    2023-06-15 11:32:59
    赞同 展开评论 打赏
  • Duplicate column name 'is_delete',应该是重复的ddl同步,可以高级设置里关闭DDL复制

    原回答者GitHub用户 agapple

    2023-06-14 22:04:18
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
PostgresChina2018_赖思超_PostgreSQL10_hash索引的WAL日志修改版final 立即下载
Kubernetes下日志实时采集、存储与计算实践 立即下载
日志数据采集与分析对接 立即下载