基于k8s安装的flink无法访问hdfs-问答-阿里云开发者社区-阿里云

开发者社区> 问答> 正文

基于k8s安装的flink无法访问hdfs

游客v4spwg6razmti 2021-09-27 17:19:27 49

flink基于k8s安装,hadoop原生安装集群,将任务放到flink上执行的时候,报错不能使用checkpoint,详细报错如下: Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded. For a full list of supported file systems, please see https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/. at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:530) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.state.filesystem.FsCheckpointStorageAccess. (FsCheckpointStorageAccess.java:64) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:527) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.StreamTask. (StreamTask.java:337) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.StreamTask. (StreamTask.java:304) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask. (SourceStreamTask.java:76) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask. (SourceStreamTask.java:72) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_292] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_292] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_292] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_292] at org.apache.flink.runtime.taskmanager.Task.loadAndInstantiateInvokable(Task.java:1515) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:727) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292] Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies. at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:55) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.state.filesystem.FsCheckpointStorageAccess. (FsCheckpointStorageAccess.java:64) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:527) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.StreamTask. (StreamTask.java:337) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.StreamTask. (StreamTask.java:304) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask. (SourceStreamTask.java:76) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask. (SourceStreamTask.java:72) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_292] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_292] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_292] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_292] at org.apache.flink.runtime.taskmanager.Task.loadAndInstantiateInvokable(Task.java:1515) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:727) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]

请求大佬看一下,主要是这两个集群,一个是k8s安装的,一个不是。

link安装 k8s安装 flink安装 无法flink flink访问
分享到
取消 提交回答
全部回答(0)
大数据
使用钉钉扫一扫加入圈子
+ 订阅

大数据计算实践乐园,近距离学习前沿技术

相似问题
最新问题