canal-adapter运行一段时间后就报错
运行一段时候后就报如下错误，重启就好了，过段时间又报错，一直这样，有时候几十分钟有时几个小时或者几天 开始用1.1.4，后来换了1.1.5还是一样
2021-03-08 14:16:06.400 [pool-4-thread-1] ERROR c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - java.lang.NullPointerException java.lang.RuntimeException: java.lang.NullPointerException at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.sync(ESSyncService.java:116) ~[na:na] at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.sync(ESSyncService.java:64) ~[na:na] at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.sync(ESAdapter.java:115) ~[na:na] at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.sync(ESAdapter.java:94) ~[na:na] at com.alibaba.otter.canal.adapter.launcher.loader.AdapterProcessor.batchSync(AdapterProcessor.java:139) ~[client-adapter.launcher-1.1.5-SNAPSHOT.jar:na] at com.alibaba.otter.canal.adapter.launcher.loader.AdapterProcessor.lambda$null$1(AdapterProcessor.java:97) ~[client-adapter.launcher-1.1.5-SNAPSHOT.jar:na] at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:891) ~[na:1.8.0_181] at com.alibaba.otter.canal.adapter.launcher.loader.AdapterProcessor.lambda$null$2(AdapterProcessor.java:94) ~[client-adapter.launcher-1.1.5-SNAPSHOT.jar:na] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_181] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181] Caused by: java.lang.NullPointerException: null at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:376) ~[na:na] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1420) ~[na:na] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1394) ~[na:na] at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:492) ~[na:na] at com.alibaba.otter.canal.client.adapter.es7x.support.ESConnection$ES7xBulkRequest.bulk(ESConnection.java:426) ~[na:na] at com.alibaba.otter.canal.client.adapter.es7x.support.ES7xTemplate.commit(ES7xTemplate.java:173) ~[na:na] at com.alibaba.otter.canal.client.adapter.es7x.support.ES7xTemplate.commitBulk(ES7xTemplate.java:343) ~[na:na] at com.alibaba.otter.canal.client.adapter.es7x.support.ES7xTemplate.update(ES7xTemplate.java:107) ~[na:na] at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.singleTableSimpleFiledUpdate(ESSyncService.java:827) ~[na:na] at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.update(ESSyncService.java:266) ~[na:na] at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.sync(ESSyncService.java:101) ~[na:na] ... 11 common frames omitted
原提问者GitHub用户andery119
写回答
-
两个解决方案：
1、你用https://github.com/alibaba/canal/pull/3923里面的代码打个包 替换你自己的版本
2、你可以给你的每个adapter都配置上key，application.yml和具体的adapter-config.yml 对应上
1、application.yml在配置adapter时指定key，如：
canalAdapters: - instance: posting_post # canal instance Name or mq topic name groups: - groupId: canal-adapter-be outerAdapters: - name: es7 key: uniq-key-1 hosts: xxx:9200 # 127.0.0.1:9200 for rest mode properties: mode: rest security.auth: elastic:pass # only used for rest mode cluster.name: elasticsearch
2、对应的posting_post.yml配置里也指定相同key，如：
key:uniq-key-1 dataSourceKey: postDS destination: posting_post groupId: canal-adapter-be esMapping: _index: index_post _id: id sql: "select xxx from post" etlCondition: "where upd_tm>={}" commitBatch: 3000
这样在canal-adapter启动时每个配置在application.yml里的canalAdapter就会各自加载属于自己的OuterAdapter实例 而不会共享同一个OuterAdapter实例，导致多线程问题
原回答者GitHub用户zhuchao941
2023-04-28 13:39:36赞同 展开评论 打赏
版权声明：本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
