开发者社区> 问答> 正文

canal 同步到 es (单机)修改 yml文件无法正常同步数据

我碰到一个问题,当修改了es/test.yml 文件的sql的查询语句改变之后,ESConfigMonitor 监听到了变化(打印日志了),但是变化之后的数据就监听不到了,无法写入es,必须重启服务?

修改之前的sql为:select t.id, t.name,t.updated_dt from test t 修改之后的sql为:select t.id, t.name from test t

日志:

2019-07-16 16:40:41.193 [pool-2-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":22,"name":"22","updated_dt":1563266438000}],"database":"test","destination":"example","es":1563266440000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1563266441172,"type":"INSERT"} 2019-07-16 16:40:41.233 [pool-2-thread-1] DEBUG c.a.otter.canal.client.adapter.es.service.ESSyncService - DML: {"data":[{"id":22,"name":"22","updated_dt":1563266438000}],"database":"test","destination":"example","es":1563266440000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1563266441194,"type":"INSERT"} Affected indexes: test 2019-07-16 16:41:36.439 [pool-2-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":23,"name":"23","updated_dt":1563266493000}],"database":"test","destination":"example","es":1563266496000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1563266496439,"type":"INSERT"} 2019-07-16 16:41:36.440 [pool-2-thread-1] DEBUG c.a.otter.canal.client.adapter.es.service.ESSyncService - DML: {"data":[{"id":23,"name":"23","updated_dt":1563266493000}],"database":"test","destination":"example","es":1563266496000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1563266496440,"type":"INSERT"} Affected indexes: test 2019-07-16 16:42:02.044 [Thread-3] INFO c.a.o.canal.client.adapter.es.monitor.ESConfigMonitor - Change a es mapping config: test.yml of canal adapter 2019-07-16 16:42:28.122 [pool-2-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":24,"name":"24","updated_dt":null}],"database":"test","destination":"example","es":1563266547000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1563266548121,"type":"INSERT"}

这时候新增的数据都是监听不到,也没有日志打印。

重启 adapter服务后报错:

2019-07-16 16:57:28.028 [pool-2-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":25,"name":"25","updated_dt":null}],"database":"test","destination":"example","es":1563266701000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1563267447953,"type":"INSERT"} 2019-07-16 16:57:28.029 [pool-2-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":26,"name":"26","updated_dt":1563267267000}],"database":"test","destination":"example","es":1563267269000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1563267447953,"type":"INSERT"} 2019-07-16 16:57:28.029 [pool-2-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":27,"name":"27","updated_dt":1563267425000}],"database":"test","destination":"example","es":1563267427000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1563267447975,"type":"INSERT"} 2019-07-16 16:57:28.031 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es.service.ESSyncService - sync error, es index: test, DML : Dml{destination='example', database='test', table='test', type='INSERT', es=1563266701000, ts=1563267448030, sql='', data=[{id=25, name=25, updated_dt=null}], old=null} 2019-07-16 16:57:28.033 [pool-2-thread-1] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{WtCn8A9jQgKdpfmojdQIgg}{127.0.0.1}{127.0.0.1:9300}]] java.lang.RuntimeException: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{WtCn8A9jQgKdpfmojdQIgg}{127.0.0.1}{127.0.0.1:9300}]] at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:110) at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:58) at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:170) at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:149) at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.batchSync(AbstractCanalAdapterWorker.java:198) at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$1(AbstractCanalAdapterWorker.java:62) at java.util.ArrayList.forEach(Unknown Source) at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$2(AbstractCanalAdapterWorker.java:58) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{WtCn8A9jQgKdpfmojdQIgg}{127.0.0.1}{127.0.0.1:9300}] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245) at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60) at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:371) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394) at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:706) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46) at com.alibaba.otter.canal.client.adapter.es.support.ESTemplate.getEsType(ESTemplate.java:499) at com.alibaba.otter.canal.client.adapter.es.support.ESTemplate.getValFromData(ESTemplate.java:347) at com.alibaba.otter.canal.client.adapter.es.support.ESTemplate.getESDataFromDmlData(ESTemplate.java:378) at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.singleTableSimpleFiledInsert(ESSyncService.java:433) at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.insert(ESSyncService.java:133) at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:93) ... 11 common frames omitted 2019-07-16 16:57:28.035 [Thread-4] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - Outer adapter sync failed! Error sync but ACK!

这时候可以监听到包括之前的数据,但是无法写入es,还必须重启es服务,才能正常工作,请问这是什么原因导致的?

原提问者GitHub用户jinkun2014

展开
收起
数据大拿 2023-05-04 11:45:46 85 0
1 条回答
写回答
取消 提交回答
  • 更新最新版试试

    原回答者GitHub用户rewerma

    2023-05-05 10:19:39
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
低代码开发师(初级)实战教程 立即下载
冬季实战营第三期:MySQL数据库进阶实战 立即下载
阿里巴巴DevOps 最佳实践手册 立即下载