env: canal 1.1.5; mysql 5.7.32; jdk1.8 场景: 从mysql同步到mysql 配置描述:有多个表需要同步,在adapter的rdb目录下配了,一个表写一个yml文件
问题1: application.yml中 key: mysql2的 threads: 3 与 rdb mytest_2.yml配置 commitBatch: 1000 是怎么样的作用,INSERT频繁是否会导致数据丢失情况,我是否需要更改这两个值
问题2:是否是那里的配置不对引发的数据无法同步
adapter.log截取
2022-11-08 20:06:59.281 [pool-73-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":6824634,"device_id":7856712}],"database":"xxx","destination":"example","es":1667909218000,"groupId":"g1","isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"abcd","ts":1667909219281,"type":"INSERT"}
application.yml配置:
server: port: 8081 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null
canal.conf: mode: tcp #tcp kafka rocketMQ rabbitMQ flatMessage: true zookeeperHosts: syncBatchSize: 1000 retries: 30 timeout: accessKey: secretKey: consumerProperties: canal.tcp.server.host: 127.0.0.1:11111 canal.tcp.batch.size: 500 canal.tcp.username: canal.tcp.password:
srcDataSources: defaultDS: url: jdbc:mysql://xxx:3306/asdf?useUnicode=true username: asdf password: asdf canalAdapters: - instance: example # canal instance Name or mq topic name groups: - groupId: g1 outerAdapters: - name: logger - name: rdb key: mysql1 properties: jdbc.driverClassName: com.mysql.jdbc.Driver jdbc.url: jdbc:mysql://xxx:3306/asdf?useUnicode=true jdbc.username: asdf jdbc.password: asdf - name: rdb key: mysql2 properties: jdbc.driverClassName: com.mysql.jdbc.Driver jdbc.url: jdbc:mysql://xxx:3306/asdf?useUnicode=true jdbc.username: asdf jdbc.password: asdf threads: 3
rdb mytest_2.yml配置:
dataSourceKey: defaultDS destination: example groupId: g1 outerAdapterKey: mysql2 concurrent: true dbMapping: database: xxx table: abcd targetTable: abcd targetPk: id: id mapAll: true
etlCondition: "where c_time>={}" readBatch: 1000 commitBatch: 1000 # 批量提交的大小
#dataSourceKey: defaultDS #destination: example #groupId: g1 #outerAdapterKey: mysql1 #concurrent: true #dbMapping:
原提问者GitHub用户chenglutao
源码bug引发的 业务需求,表字段有唯一索引,binlog同步过来的顺序为delete,insert。源码默认线程数量为3,因此分析后得出,,delete的commit还未提交insert的commit就提交了,所以造成了sql异常,,dml数据正常打印了仅此而已没有异常提示, 解决办法:
代码出处:
https://github.com/alibaba/canal/blob/master/client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/service/RdbSyncService.java
原回答者GitHub用户chenglutao
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。