开发者社区> 问答> 正文

canal全量同步大量数据有丢失的情况

主表有430万数据,关联表有220万数据,是多对一的关系,理论上同步后会有430万数据。

实际上:

数据全量导入完成, 一共导入 4134977 条数据, 耗时: 7995600

过程中有报错如下:

2021-11-17 02:11:21.710 [pool-3-thread-4] ERROR com.alibaba.otter.canal.client.adapter.support.Util - sqlRs has error, sql: 这里是我的SQL LIMIT 4240000,10000 2021-11-17 02:11:21.711 [pool-3-thread-4] ERROR c.a.otter.canal.client.adapter.es7x.etl.ESEtlService - com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 3, maxActive 3, creating 0 java.lang.RuntimeException: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 3, maxActive 3, creating 0 at com.alibaba.otter.canal.client.adapter.support.Util.sqlRS(Util.java:65) ~[client-adapter.common-1.1.5.jar:na] at com.alibaba.otter.canal.client.adapter.es7x.etl.ESEtlService.executeSqlImport(ESEtlService.java:64) ~[na:na] at com.alibaba.otter.canal.client.adapter.support.AbstractEtlService.lambda$importData$1(AbstractEtlService.java:91) [client-adapter.common-1.1.5.jar:na] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_221] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_221] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_221] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_221] Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 3, maxActive 3, creating 0 at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1749) ~[druid-1.2.6.jar:1.2.6] at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1419) ~[druid-1.2.6.jar:1.2.6] at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1399) ~[druid-1.2.6.jar:1.2.6] at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1389) ~[druid-1.2.6.jar:1.2.6] at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:100) ~[druid-1.2.6.jar:1.2.6] at com.alibaba.otter.canal.client.adapter.support.Util.sqlRS(Util.java:50) ~[client-adapter.common-1.1.5.jar:na] ... 6 common frames omitted

全量同步的过程中有数据的更新及插入操作。有没有办法解决这个问题呢?

原提问者GitHub用户geekerstar

展开
收起
山海行 2023-04-27 11:22:12 126 0
1 条回答
写回答
取消 提交回答
  • com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 3, maxActive 3,连接池开的太小

    原回答者GitHub用户agapple

    2023-04-27 21:03:54
    赞同 展开评论 打赏
问答分类:
问答地址:
问答排行榜
最热
最新

相关电子书

更多
低代码开发师(初级)实战教程 立即下载
冬季实战营第三期:MySQL数据库进阶实战 立即下载
阿里巴巴DevOps 最佳实践手册 立即下载