开发者社区 > 大数据与机器学习 > 实时计算 Flink > 正文

Flink CDC里不同机器的TaskManagerRunner,如何指定端口?

Flink CDC里不同机器的TaskManagerRunner,如何指定端口,我看现在的端口是随机的,现在的端口都是这种随机的。1445ea0f75afe924c6ab7785de5048bd.png2024-04-26 12:09:39,298 WARN org.apache.flink.runtime.jobmaster.slotpool.DeclarativeSlotPoolBridge [] - Could not acquire the minimum required resources, failing slot requests. Acquired: [ResourceRequirement{resourceProfile=ResourceProfile{cpuCores=1, taskHeapMemory=1009.200mb (1058222878 bytes), taskOffHeapMemory=0 bytes, managedMemory=1012.160mb (1061326698 bytes), networkMemory=253.040mb (265331674 bytes)}, numberOfRequiredSlots=1}]. Current slot pool status: Registered TMs: 1, registered slots: 1 free slots: 0
2024-04-26 12:09:39,303 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Flink CDC Event Source: mysql -> SchemaOperator -> PrePartition (1/2) (0338d48bfd91255840887a522cfcfb39_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from SCHEDULED to FAILED on [unassigned resource].
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not acquire the minimum required resources.资源不够吗?2024-04-26 12:29:14,185 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Flink CDC Event Source: mysql -> SchemaOperator -> PrePartition (2/2) (6c19762a48b678d316854d51c1e698a2_cbc357ccb763df2852fee8c4fc7d55f2_1_0) switched from RUNNING to FAILED on 172.21.48.3:41475-2155b8 @ VM-48-3-tencentos (dataPort=41717).
org.apache.flink.util.SerializedThrowable: java.lang.IllegalStateException: Failed to send request to coordinator: com.ververica.cdc.runtime.operators.schema.event.SchemaChangeRequest@4a3ec56c
at com.ververica.cdc.runtime.operators.schema.SchemaOperator.sendRequestToCoordinator(SchemaOperator.java:126) ~[?:?]
at com.ververica.cdc.runtime.operators.schema.SchemaOperator.requestSchemaChange(SchemaOperator.java:110) ~[?:?]
at com.ververica.cdc.runtime.operators.schema.SchemaOperator.handleSchemaChangeEvent(SchemaOperator.java:95) ~[?:?]
at com.ververica.cdc.runtime.operators.schema.SchemaOperator.processElement(SchemaOperator.java:85) ~[?:?]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:309) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:101) ~[flink-dist-1.18.0.jar:1.18.0]
at com.ververica.cdc.connectors.mysql.source.reader.MySqlPipelineRecordEmitter.sendCreateTableEvent(MySqlPipelineRecordEmitter.java:125) ~[?:?]
at com.ververica.cdc.connectors.mysql.source.reader.MySqlPipelineRecordEmitter.processElement(MySqlPipelineRecordEmitter.java:109) ~[?:?]
at com.ververica.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitRecord(MySqlRecordEmitter.java:82) ~[?:?]
at com.ververica.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitRecord(MySqlRecordEmitter.java:55) ~[?:?]
at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:160) ~[flink-connector-files-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:419) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) ~[flink-dist-1.18.0.jar:1.18.0]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_391]
Caused by: org.apache.flink.util.SerializedThrowable: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Invocation of [RemoteRpcInvocation(JobMasterOperatorEventGateway.sendRequestToCoordinator(OperatorID, SerializedValue))] at recipient [pekko.tcp://flink@172.21.48.3:6123/user/rpc/jobmanager_2] timed out. This is usually caused by: 1) Pekko failed sending the message silently, due to problems like oversized payload or serialization failures. In that case, you should find detailed error information in the logs. 2) The recipient needs more time for responding, due to problems like slow machines or network jitters. In that case, you can try to increase pekko.ask.timeout.

展开
收起
小小鹿鹿鹿 2024-05-01 08:55:48 15 0
1 条回答
写回答
取消 提交回答
  • 看起来是 JobManager 和 TaskManager 之间的通信连接超时了。默认状态下超时时间设置是 10 秒钟,您可以尝试设置一下 Flink 配置文件的 pekko.ask.timeout 参数看看会不会缓解问题?此回答来自钉群Flink CDC 社区 。

    2024-05-01 13:03:29
    赞同 1 展开评论 打赏

实时计算Flink版是阿里云提供的全托管Serverless Flink云服务,基于 Apache Flink 构建的企业级、高性能实时大数据处理系统。提供全托管版 Flink 集群和引擎,提高作业开发运维效率。

相关电子书

更多
Flink CDC Meetup PPT - 龚中强 立即下载
Flink CDC Meetup PPT - 王赫 立即下载
Flink CDC Meetup PPT - 覃立辉 立即下载