开发者社区> 问答> 正文

为什么yarn.provided.lib.dirs在flink1.11 yarn提交不生效?

flink1.11 on yarn模式,我提前将flink  lib下的依赖及自定义函数jar上传到hdfs上,提交时使用yarn.provided.lib.dirs  指定hdfs的依赖路径。原本设想程序中使用反射去寻找自定义函数的类并且实例化,但是提交时报错,程序并没有找到自定义函数的路径 

提交命令:/usr/hdp/flink1.11/bin/flink run -m yarn-cluster -d -ynm udf-test -yD  yarn.provided.lib.dirs=hdfs://ip:8020/flink-yarn/jars -c  com.ly.common.udf.demo.FlinkUDFDemo /data/bigdata/jars/udf-test.jar 

相关信息如下:  2020-12-22 08:41:11,157 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli   [] - Dynamic Property set:  yarn.provided.lib.dirs=hdfs://chaitin/flink-yarn/jars  2020-12-22 08:41:11,157 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli   [] - Dynamic Property set:  yarn.provided.lib.dirs=hdfs://chaitin/flink-yarn/jars  -- class path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 

------------------------------------------------------------  The program finished with the following exception: 

org.apache.flink.client.program.ProgramInvocationException: The main method  caused an error: object com.ly.third.udf.flink.SortKey not found.  at  org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)  at  org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)  at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)  at  org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:699)  at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:232)  at  org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916)  at  org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992)  at java.security.AccessController.doPrivileged(Native Method)  at javax.security.auth.Subject.doAs(Subject.java:422)  at  org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)  at  org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)  at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)  Caused by: scala.ScalaReflectionException: object  com.ly.third.udf.flink.SortKey not found.  at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:162)  at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:22)  at  com.ly.common.udf.reflect.RegisterFlinkFunction$.loadFlinkFunction(RegisterFlinkFunction.scala:14)  at com.ly.common.udf.demo.FlinkUDFDemo$.main(FlinkUDFDemo.scala:27)  at com.ly.common.udf.demo.FlinkUDFDemo.main(FlinkUDFDemo.scala)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at  sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  at  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  at java.lang.reflect.Method.invoke(Method.java:498)  at  org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)  ... 11 more*来自志愿者整理的flink邮件归档

展开
收起
CCCC 2021-12-02 14:44:36 1714 0
1 条回答
写回答
取消 提交回答
  • 使用 -Dyarn.provided.lib.dirs 试试*来自志愿者整理的FLINK邮件归档

    2021-12-02 16:06:31
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
深度学习+大数据 TensorFlow on Yarn 立即下载
Docker on Yarn 微服务实践 立即下载
深度学习+大数据-TensorFlow on Yarn 立即下载