canal1.1.0 并发解析下多线程对GTID更新操作,导致java.util.Concurren

具体错误如下:

2018-09-12 18:38:25,776||destination = 1002 , address = /*********:3306 , EventParser|?|ERROR|c.a.o.c.p.i.mysql.MysqlEventParser - dump address ************:3306 has an error, retrying. caused by com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: java.util.ConcurrentModificationException: null at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) ~[na:1.8.0_121] at java.util.ArrayList$Itr.next(ArrayList.java:851) ~[na:1.8.0_121] at com.alibaba.otter.canal.parse.driver.mysql.packets.UUIDSet.toString(UUIDSet.java:125) ~[canal.parse.driver-1.1.0-20180912.102450-4.jar:na] at com.alibaba.otter.canal.parse.driver.mysql.packets.MysqlGTIDSet.toString(MysqlGTIDSet.java:111) ~[canal.parse.driver-1.1.0-20180912.102450-4.jar:na] at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.createHeader(LogEventConvert.java:849) ~[canal.parse-1.1.0-20180912.102500-4.jar:na] at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEvent(LogEventConvert.java:561) ~[canal.parse-1.1.0-20180912.102500-4.jar:na] at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$DmlParserStage.onEvent(MysqlMultiStageCoprocessor.java:302) ~[canal.parse-1.1.0-20180912.102500-4.jar:na] at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$DmlParserStage.onEvent(MysqlMultiStageCoprocessor.java:288) ~[canal.parse-1.1.0-20180912.102500-4.jar:na] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) ~[disruptor-3.4.2.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] 2018-09-12 18:38:25,776||destination = 1002 , address = /172.16.2.71:3306 , EventParser|?|ERROR|c.a.o.c.common.alarm.LogAlarmHandler - destination:1002[com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) at java.util.ArrayList$Itr.next(ArrayList.java:851) at com.alibaba.otter.canal.parse.driver.mysql.packets.UUIDSet.toString(UUIDSet.java:125) at com.alibaba.otter.canal.parse.driver.mysql.packets.MysqlGTIDSet.toString(MysqlGTIDSet.java:111) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.createHeader(LogEventConvert.java:849) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEvent(LogEventConvert.java:561) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$DmlParserStage.onEvent(MysqlMultiStageCoprocessor.java:302) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$DmlParserStage.onEvent(MysqlMultiStageCoprocessor.java:288) at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ]st.java:901) at java.util.ArrayList$Itr.next(ArrayList.java:851) at com.alibaba.otter.canal.parse.driver.mysql.packets.UUIDSet.toString(UUIDSet.java:125) at com.alibaba.otter.canal.parse.driver.mysql.packets.MysqlGTIDSet.toString(MysqlGTIDSet.java:111) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.createHeader(LogEventConvert.java:849) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEvent(LogEventConvert.java:561) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$DmlParserStage.onEvent(MysqlMu

JDK: 1.8.0_21

配置信息:

preallell=true parallelthreadSize=4(计算结果)

分析原因(不一定正确): 在LogEventConvert的180行,对gtid_set进行了更新,其实是更新了MysqlGTIDSet的34行, sets.get(sid).intervals.addAll(us.intervals),这个地方更新了list的modCount++, 在UUIDSet的tostring方法中,遍历了intervals,结果触发了ArrayList的901行的checkForComodification方法, 由于addAll增加了modCount,但是execptedModCount没有增加,跑出了异常.

解决方法:

方法1:preallell=false 方法2:多线程情况下,是不是需要加锁.

原提问者GitHub用户mycat-lulin