开发者社区> 问答> 正文

flink yarn application 提交任务出错

今天尝试使用yarn application模式(带yarn.provided.lib.dirs参数),将$FLINK_LIB目录下的jar包上传到了hdfs,结果报了如下的错,是少了哪个jar包或配置文件吗?

org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn Application Cluster

at org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:443)

at com.huifu.streamsql.launcher.yarn.YarnApplicationExecutor.doStart(YarnApplicationExecutor.java:70)

at com.huifu.streamsql.launcher.AbstractJobExecutor.start(AbstractJobExecutor.java:76)

at com.huifu.streamsql.launcher.JobCommand$1.execute(JobCommand.java:12)

at com.huifu.streamsql.launcher.SubmitJobMain.main(SubmitJobMain.java:39)

Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:9000/flinkLib, expected: file:///

at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:665)

at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:86)

at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:630)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:861)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:625)

at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1437)

at org.apache.flink.yarn.YarnApplicationFileUploader.lambda$getAllFilesInProvidedLibDirs$2(YarnApplicationFileUploader.java:429)

at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedConsumer$3(FunctionUtils.java:94)

at java.util.ArrayList.forEach(ArrayList.java:1259)

at org.apache.flink.yarn.YarnApplicationFileUploader.getAllFilesInProvidedLibDirs(YarnApplicationFileUploader.java:426)

at org.apache.flink.yarn.YarnApplicationFileUploader. (YarnApplicationFileUploader.java:109)

at org.apache.flink.yarn.YarnApplicationFileUploader.from(YarnApplicationFileUploader.java:354)

at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:710)

at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:558)

at org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:436)

... 4 more*来自志愿者整理的flink邮件归档

展开
收起
JACKJACK 2021-12-08 16:13:06 1872 0
1 条回答
写回答
取消 提交回答
  • 对的是我!

    这个报错应该是你HDFS的core-site.xml的配置有问题 defaultFS应该是hdfs://localhost:9000/才对,你可以检查一下*来自志愿者整理的flink邮件归档

    2021-12-08 16:54:00
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
深度学习+大数据 TensorFlow on Yarn 立即下载
Docker on Yarn 微服务实践 立即下载
深度学习+大数据-TensorFlow on Yarn 立即下载