开发者社区> 问答> 正文

日志有完整DML内容,但是INSERT数据有时会出现丢失没同步

env: canal 1.1.5; mysql 5.7.32; jdk1.8 场景: 从mysql同步到mysql 配置描述:有多个表需要同步,在adapter的rdb目录下配了,一个表写一个yml文件

问题1: application.yml中 key: mysql2的 threads: 3 与 rdb mytest_2.yml配置 commitBatch: 1000 是怎么样的作用,INSERT频繁是否会导致数据丢失情况,我是否需要更改这两个值

问题2:是否是那里的配置不对引发的数据无法同步

adapter.log截取

2022-11-08 20:06:59.281 [pool-73-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":6824634,"device_id":7856712}],"database":"xxx","destination":"example","es":1667909218000,"groupId":"g1","isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"abcd","ts":1667909219281,"type":"INSERT"}

application.yml配置:

server: port: 8081 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null

canal.conf: mode: tcp #tcp kafka rocketMQ rabbitMQ flatMessage: true zookeeperHosts: syncBatchSize: 1000 retries: 30 timeout: accessKey: secretKey: consumerProperties: canal.tcp.server.host: 127.0.0.1:11111 canal.tcp.batch.size: 500 canal.tcp.username: canal.tcp.password:

srcDataSources: defaultDS: url: jdbc:mysql://xxx:3306/asdf?useUnicode=true username: asdf password: asdf canalAdapters: - instance: example # canal instance Name or mq topic name groups: - groupId: g1 outerAdapters: - name: logger - name: rdb key: mysql1 properties: jdbc.driverClassName: com.mysql.jdbc.Driver jdbc.url: jdbc:mysql://xxx:3306/asdf?useUnicode=true jdbc.username: asdf jdbc.password: asdf - name: rdb key: mysql2 properties: jdbc.driverClassName: com.mysql.jdbc.Driver jdbc.url: jdbc:mysql://xxx:3306/asdf?useUnicode=true jdbc.username: asdf jdbc.password: asdf threads: 3

rdb mytest_2.yml配置:

dataSourceKey: defaultDS destination: example groupId: g1 outerAdapterKey: mysql2 concurrent: true dbMapping: database: xxx table: abcd targetTable: abcd targetPk: id: id mapAll: true

caseInsensitive: true

targetColumns:

id:

name:

role_id:

c_time:

test1:

etlCondition: "where c_time>={}" readBatch: 1000 commitBatch: 1000 # 批量提交的大小

Mirror schema synchronize config

#dataSourceKey: defaultDS #destination: example #groupId: g1 #outerAdapterKey: mysql1 #concurrent: true #dbMapping:

mirrorDb: true

database: mytest

原提问者GitHub用户chenglutao

展开
收起
白夜行fighting 2023-04-25 20:24:36 221 0
1 条回答
写回答
取消 提交回答
  • 源码bug引发的 业务需求,表字段有唯一索引,binlog同步过来的顺序为delete,insert。源码默认线程数量为3,因此分析后得出,,delete的commit还未提交insert的commit就提交了,所以造成了sql异常,,dml数据正常打印了仅此而已没有异常提示, 解决办法:

    回答3.png

    代码出处:

    https://github.com/alibaba/canal/blob/master/client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/service/RdbSyncService.java

    回答4.png

    原回答者GitHub用户chenglutao

    2023-04-26 17:16:56
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
PostgresChina2018_赖思超_PostgreSQL10_hash索引的WAL日志修改版final 立即下载
Kubernetes下日志实时采集、存储与计算实践 立即下载
日志数据采集与分析对接 立即下载