开发者社区> 问答> 正文

canal读取binlog同步数据到ES增量更新不好使,有没有懂的帮看看

canal.adapter控制台可以打印出dml日志,但就是同步不到ES,我在mysql数据库做增删改查操作数据都可以读取到日志,但就是不会同步到ES索引,下面是application.yml配置 :

server: port: 8081 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null

canal.conf: mode: tcp #tcp kafka rocketMQ rabbitMQ flatMessage: true zookeeperHosts: syncBatchSize: 1000 retries: -1 timeout: accessKey: secretKey: consumerProperties:

canal tcp consumer

canal.tcp.server.host: 127.0.0.1:11111 canal.tcp.zookeeper.hosts: canal.tcp.batch.size: 500 canal.tcp.username: canal.tcp.password:

kafka consumer

kafka.bootstrap.servers: 127.0.0.1:9092 kafka.enable.auto.commit: false kafka.auto.commit.interval.ms: 1000 kafka.auto.offset.reset: latest kafka.request.timeout.ms: 40000 kafka.session.timeout.ms: 30000 kafka.isolation.level: read_committed kafka.max.poll.records: 1000

rocketMQ consumer

rocketmq.namespace: rocketmq.namesrv.addr: 127.0.0.1:9876 rocketmq.batch.size: 1000 rocketmq.enable.message.trace: false rocketmq.customized.trace.topic: rocketmq.access.channel: rocketmq.subscribe.filter:

rabbitMQ consumer

rabbitmq.host: rabbitmq.virtual.host: rabbitmq.username: rabbitmq.password: rabbitmq.resource.ownerId:

srcDataSources: pprsPermitDS: url: jdbc:mysql://127.0.0.1:3306/canal-test?useUnicode=true username: canal password: canal canalAdapters:

instance: example # canal instance Name or mq topic name groups: groupId: g1 outerAdapters: name: logger name: es7 hosts: http://127.0.0.1:9200 properties: mode: rest cluster.name: elasticsearch 下面是es7目录下的映射文件product.yml :

dataSourceKey: pprsPermitDS destination: example groupId: g1 esMapping: _index: canalproduct _id: _id _type: doc upsert: true sql: "SELECT p.id as _id, p.title, p.subtitle, p.pic FROMproduct p" commitBatch: 3000

下面是canal.deployer/conf/example/instance.properties配置:

canal.instance.gtidon=false

canal.instance.master.address=127.0.0.1:3306 canal.instance.master.journal.name= canal.instance.master.position= canal.instance.master.timestamp= canal.instance.master.gtid=

canal.instance.rds.accesskey= canal.instance.rds.secretkey= canal.instance.rds.instanceId=

canal.instance.defaultDatabaseName = canal-test

canal.instance.dbUsername=canal canal.instance.dbPassword=canal canal.instance.connectionCharset = UTF-8

canal.instance.enableDruid=false

canal.instance.filter.regex=...

canal.instance.filter.black.regex=

canal.mq.topic=example canal.mq.partition=0

原提问者GitHub用户haifengkok

展开
收起
白夜行fighting 2023-04-25 20:44:39 346 0
1 条回答
写回答
取消 提交回答
  • 是因为在es7目录查询语句那个yml里的 表名 带了单引号,把引号去掉就能同步了

    原回答者GitHub用户haifengkok

    2023-04-26 17:27:36
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
PolarDB-X 2.0 全局 Binlog 与备份恢复能 立即下载
低代码开发师(初级)实战教程 立即下载
阿里巴巴DevOps 最佳实践手册 立即下载