flink 1.19.1 standalone模式环境,资源充足,启动任何一个job后,rocksdb以及checkpoint存储路径指向挂载的外部存储时,checkpoint失败,使用节点系统中的存储时checkpoint正常。
挂载的存储权限可读可写可执行,节点存储挂载方式:mount -t cifs -o username=xs,password=disk@001,rw,dir_mode=0777,file_mode=0777 //172.16.152.12/xs-dir/27b/disk01
报错如下:
jobmanager日志:
2025-06-11 18:45:20,611 INFO org.apache.flink.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives notice from subtask 0 for the stream split assignment.
2025-06-11 18:45:24,366 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 1 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1749638724359 for job a6a9536566212af636a0fbb7480760b8.
2025-06-11 18:45:24,504 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Decline checkpoint 1 by task a2473bc060d2bb703c00ff09e1a310ae_e03d9229619566fd5d62dea3a8d25d1e_0_0 of job a6a9536566212af636a0fbb7480760b8 at 10.42.5.43:6122-5b1838 @ 10-42-5-43.service-flink-taskmanager-service.mah.svc.cluster.local (dataPort=36841).
org.apache.flink.util.SerializedThrowable: org.apache.flink.runtime.checkpoint.CheckpointException: Asynchronous task checkpoint failed.
at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.handleExecutionException(AsyncCheckpointRunnable.java:320) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:155) ~[flink-dist-1.19.1.jar:1.19.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_412]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_412]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_412]
Caused by: org.apache.flink.util.SerializedThrowable: java.lang.Exception: Could not materialize checkpoint 1 for operator ChangelogNormalize[284] -> Correlate[285] -> Calc[286] -> LocalGroupAggregate[287] (1/1)#0.
at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.handleExecutionException(AsyncCheckpointRunnable.java:298) ~[flink-dist-1.19.1.jar:1.19.1]
... 4 more
Caused by: org.apache.flink.util.SerializedThrowable: java.util.concurrent.ExecutionException: java.nio.file.FileSystemException: /opt/flink/data/flink-rocksdb/job_a6a9536566212af636a0fbb7480760b8_op_KeyedMapBundleOperator_e03d9229619566fd5d62dea3a8d25d1e1_1uuid_c328310a-02d0-45a8-822a-4397c162eb69/chk-1/000010.sst: Invalid argument
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_412]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_412]
at org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:511) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:54) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.finalizeNonFinishedSnapshots(AsyncCheckpointRunnable.java:191) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:124) ~[flink-dist-1.19.1.jar:1.19.1]
... 3 more
Caused by: org.apache.flink.util.SerializedThrowable: java.nio.file.FileSystemException: /opt/flink/data/flink-rocksdb/job_a6a9536566212af636a0fbb7480760b8_op_KeyedMapBundleOperator_e03d9229619566fd5d62dea3a8d25d1e1_1uuid_c328310a-02d0-45a8-822a-4397c162eb69/chk-1/000010.sst: Invalid argument
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:1.8.0_412]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_412]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_412]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:1.8.0_412]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_412]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_412]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_412]
at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_412]
at org.apache.flink.contrib.streaming.state.RocksDBStateUploader.uploadLocalFileToCheckpointFs(RocksDBStateUploader.java:135) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.contrib.streaming.state.RocksDBStateUploader.lambda$null$0(RocksDBStateUploader.java:111) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:32) ~[flink-dist-1.19.1.jar:1.19.1]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ~[?:1.8.0_412]
... 3 more
2025-06-11 18:45:24,515 WARN org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 1 for job a6a9536566212af636a0fbb7480760b8. (0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Asynchronous task checkpoint failed.
at org.apache.flink.runtime.messages.checkpoint.SerializedCheckpointException.unwrap(SerializedCheckpointException.java:51) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:1138) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$declineCheckpoint$3(ExecutionGraphHandler.java:123) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$4(ExecutionGraphHandler.java:139) ~[flink-dist-1.19.1.jar:1.19.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_412]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_412]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_412]
Caused by: org.apache.flink.util.SerializedThrowable: org.apache.flink.runtime.checkpoint.CheckpointException: Asynchronous task checkpoint failed.
at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.handleExecutionException(AsyncCheckpointRunnable.java:320) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:155) ~[flink-dist-1.19.1.jar:1.19.1]
... 3 more
Caused by: org.apache.flink.util.SerializedThrowable: java.lang.Exception: Could not materialize checkpoint 1 for operator ChangelogNormalize[284] -> Correlate[285] -> Calc[286] -> LocalGroupAggregate[287] (1/1)#0.
at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.handleExecutionException(AsyncCheckpointRunnable.java:298) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:155) ~[flink-dist-1.19.1.jar:1.19.1]
... 3 more
Caused by: org.apache.flink.util.SerializedThrowable: java.util.concurrent.ExecutionException: java.nio.file.FileSystemException: /opt/flink/data/flink-rocksdb/job_a6a9536566212af636a0fbb7480760b8_op_KeyedMapBundleOperator_e03d9229619566fd5d62dea3a8d25d1e1_1uuid_c328310a-02d0-45a8-822a-4397c162eb69/chk-1/000010.sst: Invalid argument
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_412]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_412]
at org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:511) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:54) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.finalizeNonFinishedSnapshots(AsyncCheckpointRunnable.java:191) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:124) ~[flink-dist-1.19.1.jar:1.19.1]
... 3 more
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。
实时计算Flink版是阿里云提供的全托管Serverless Flink云服务,基于 Apache Flink 构建的企业级、高性能实时大数据处理系统。提供全托管版 Flink 集群和引擎,提高作业开发运维效率。