开发者社区> 问答> 正文

关于flink 写于kafka时的transactionId 生成问题

hi,在写入kafka的时候自动生成了一个transactionId,请问这个id生成的方式是什么,我自己指定好像并不起作用。 作业我已经停止了,但是看kafka的日志还是不断的在刷Initialized transactionalId………. ,而且该程序再此启动就会报: Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker. 错误,请问有什么办法可以避免这个问题? 下面是kafka的日志,一直在刷,很多都是好几天之前的任务了

2019-08-29 17:08:48,911] INFO [TransactionCoordinator id=0] Initialized transactionalId Split Reader: Custom File Source -> (Sink: Print to Std. Out, Sink: Unnamed)-7b96fd3ac98d0efc878b17e02839b3d6-3 with producerId 1011 and producer epoch 14255 on partition __transaction_state-36 (kafka.coordinator.transaction.TransactionCoordinator) [2019-08-29 17:08:49,244] INFO [TransactionCoordinator id=0] Initialized transactionalId Split Reader: Custom File Source -> (Sink: Print to Std. Out, Sink: Unnamed)-7b96fd3ac98d0efc878b17e02839b3d6-4 with producerId 1012 and producer epoch 16544 on partition __transaction_state-35 (kafka.coordinator.transaction.TransactionCoordinator) [2019-08-29 17:08:49,518] INFO [TransactionCoordinator id=0] Initialized transactionalId Split Reader: Custom File Source -> (Sink: Print to Std. Out, Sink: Unnamed)-7b96fd3ac98d0efc878b17e02839b3d6-1 with producerId 1013 and producer epoch 15460 on partition __transaction_state-38 (kafka.coordinator.transaction.TransactionCoordinator) [2019-08-29 17:08:49,786] INFO [TransactionCoordinator id=0] Initialized transactionalId Split Reader: Custom File Source -> (Sink: Print to Std. Out, Sink: Unnamed)-7b96fd3ac98d0efc878b17e02839b3d6-2 with producerId 1014 and producer epoch 7781 on partition __transaction_state-37 (kafka.coordinator.transaction.TransactionCoordinator) [2019-08-29 17:08:50,054] INFO [TransactionCoordinator id=0] Initialized transactionalId Split Reader: Custom File Source -> (Sink: Print to Std. Out, Sink: Unnamed)-7b96fd3ac98d0efc878b17e02839b3d6-0 with producerId 1015 and producer epoch 7529 on partition __transaction_state-39 (kafka.coordinator.transaction.TransactionCoordinator) [2019-08-29 17:08:50,310] INFO [TransactionCoordinator id=0] Initialized transactionalId Split Reader: Custom File Source -> (Sink: Print to Std. Out, Sink: Unnamed)-7b96fd3ac98d0efc878b17e02839b3d6-3 with producerId 1011 and producer epoch 14256 on partition __transaction_state-36 (kafka.coordinator.transaction.TransactionCoordinator) [2019-08-29 17:08:51,078] INFO [TransactionCoordinator id=0] Initialized transactionalId Split Reader: Custom File Source -> (Sink: Print to Std. Out, Sink: Unnamed)-7b96fd3ac98d0efc878b17e02839b3d6-4 with producerId 1012 and producer epoch 16545 on partition __transaction_state-35 (kafka.coordinator.transaction.TransactionCoordinator) [2019-08-29 17:08:53,047] INFO [TransactionCoordinator id=0] Initialized transactionalId Split Reader: Custom File Source -> (Sink: Print to Std. Out, Sink: Unnamed)-7b96fd3ac98d0efc878b17e02839b3d6-1 with producerId 1013 and producer epoch 15461 on partition __transaction_state-38 (kafka.coordinator.transaction.TransactionCoordinator) *来自志愿者整理的flink邮件归档

展开
收起
小阿怪 2021-12-07 22:01:44 1307 0
1 条回答
写回答
取消 提交回答
  • Hi,Maybe you want to check all the broker and producers have the same timezone setup, and all time are synchronous.*来自志愿者整理的flink邮件归档

    2021-12-08 10:24:29
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
Java Spring Boot开发实战系列课程【第16讲】:Spring Boot 2.0 实战Apache Kafka百万级高并发消息中间件与原理解析 立即下载
MaxCompute技术公开课第四季 之 如何将Kafka数据同步至MaxCompute 立即下载
消息队列kafka介绍 立即下载