在AWS EMR中部署Flink自定义JAR文件时出错-问答-阿里云开发者社区-阿里云

开发者社区> flink小助手> 正文

在AWS EMR中部署Flink自定义JAR文件时出错

2018-12-06 18:06:15 4054 1

基本上我想将Flink自定义JAR文件部署到新的AWS EMR集群。以下是我所做的总结。我创建了一个新的AWS EMR集群。

第1步:软件和步骤更改 -

使用flink作为服务创建AWS EMR集群。(EMR发布版本 - 5.17.0)并单击Flink 1.5.2作为软件配置。
输入配置JSON: -

[{“Classification”:“flink-conf”,“Properties”:{“jobmanager.heap.mb”:“3072”,“taskmanager.heap.mb”:“51200”,“taskmanager.numberOfTaskSlots”:“2” ,“taskmanager.memory.preallocate”:“false”,
“parallelism.default”:“1”}]

步骤2:硬件 - 硬件配置没有变化。默认情况下,我们有1个主服务器,2个核心服务器和0个任务实例。全部都是m3.xlarge类型。

第3步:常规群集设置 - 此处无变化。
Step4:安全性 - 提供我的EC2密钥对。
一旦集群创建准备就绪,我就连接到EC2机器并尝试部署自定义jar文件。以下是我每次尝试通过CLI部署它时遇到的不同错误。

1)

flink run -m yarn-cluster -yn 2 -c com.deepak.flink.examples.WordCount flink-examples-assembly-1.0.jar

Using the result of 'hadoop classpath' to augment the Hadoop classpath: /etc/hadoop/conf:/usr/lib/hadoop/lib/:/usr/lib/hadoop/.//:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/:/usr/lib/hadoop-hdfs/.//:/usr/lib/hadoop-yarn/lib/:/usr/lib/hadoop-yarn/.//:/usr/lib/hadoop-mapreduce/lib/:/usr/lib/hadoop-mapreduce/.//::/usr/lib/hadoop-lzo/lib/:/usr/share/aws/aws-java-sdk/:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/:/usr/share/aws/emr/emrfs/auxlib/:/usr/share/aws/emr/cloudwatch-sink/lib/:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/flink/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2018-10-09 06:30:36,766 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at ip-IPADDRESS.ec2.internal/IPADDRESS:8032
2018-10-09 06:30:36,909 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2018-10-09 06:30:37,168 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Killing YARN application
2)

flink run -c com.deepak.flink.examples.WordCount flink-examples-assembly-1.0.jar

Using the result of 'hadoop classpath' to augment the Hadoop classpath: /etc/hadoop/conf:/usr/lib/hadoop/lib/:/usr/lib/hadoop/.//:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/:/usr/lib/hadoop-hdfs/.//:/usr/lib/hadoop-yarn/lib/:/usr/lib/hadoop-yarn/.//:/usr/lib/hadoop-mapreduce/lib/:/usr/lib/hadoop-mapreduce/.//::/usr/lib/hadoop-lzo/lib/:/usr/share/aws/aws-java-sdk/:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/:/usr/share/aws/emr/emrfs/auxlib/:/usr/share/aws/emr/cloudwatch-sink/lib/:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/flink/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]


The program finished with the following exception:

org.apache.flink.client.deployment.ClusterRetrieveException: Couldn't retrieve standalone cluster

at org.apache.flink.client.deployment.StandaloneClusterDescriptor.retrieve(StandaloneClusterDescriptor.java:51)
at org.apache.flink.client.deployment.StandaloneClusterDescriptor.retrieve(StandaloneClusterDescriptor.java:31)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:253)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:214)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1025)
at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1101)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1840)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1101)

Caused by: org.apache.flink.util.ConfigurationException: Config parameter 'Key: 'jobmanager.rpc.address' , default: null (deprecated keys: [])' is missing (hostname/address of JobManager to connect to).

at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.getJobManagerAddress(HighAvailabilityServicesUtils.java:141)
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:81)
at org.apache.flink.client.program.ClusterClient.<init>(ClusterClient.java:158)
at org.apache.flink.client.program.rest.RestClusterClient.<init>(RestClusterClient.java:183)
at org.apache.flink.client.program.rest.RestClusterClient.<init>(RestClusterClient.java:156)
at org.apache.flink.client.deployment.StandaloneClusterDescriptor.retrieve(StandaloneClusterDescriptor.java:49)
... 10 more

即使我尝试通过AWS Web UI进行部署,也无法部署jar。

所以,基本上我想将自定义JAR部署到flink YARN Cluster。我不确定YARN flink配置或其他任何东西我缺少什么。

取消 提交回答
全部回答(1)
  • flink小助手
    2019-07-17 23:18:39

    您应该减少任务管理器的内存分配。目前,您正在尝试分配51.2G内存,而单个m3.xlarge机器只有15G内存,而且总共30G用于2台机器集群。

    0 0
相关问答

162

回答

惊喜翻倍:免费ECS+免费环境配置~!(ECS免费体验6个月活动3月31日结束)

豆妹 2014-10-29 17:52:21 222937浏览量 回答数 162

8

回答

OceanBase 使用动画(持续更新)

mq4096 2019-02-20 17:16:36 333783浏览量 回答数 8

110

回答

OSS存储服务-客户端工具

newegg11 2012-05-17 15:37:18 292224浏览量 回答数 110

22

回答

爬虫数据管理【问答合集】

我是管理员 2018-08-10 16:37:41 146741浏览量 回答数 22

18

回答

阿里云开放端口权限

xcxx 2016-07-20 15:03:33 643008浏览量 回答数 18

33

回答

Win Server 2003-2016 加密勒索事件必打补丁合集

妙正灰 2017-05-15 10:44:38 277458浏览量 回答数 33

38

回答

安全组详解,新手必看教程

我的中国 2017-11-30 15:23:46 256825浏览量 回答数 38

294

回答

Linux Bash严重漏洞修复紧急通知(已全部给出最终修复方案)

qilu 2014-09-25 13:26:50 431576浏览量 回答数 294

24

回答

【精品问答】python技术1000问(1)

问问小秘 2019-11-15 13:25:00 471184浏览量 回答数 24

13

回答

游戏云精彩帖汇总

nono20011908 2014-08-22 11:00:12 203607浏览量 回答数 13
+关注
flink小助手
flink小助手会定期更新直播回顾等资料和文章干货,还整合了大家在钉群提出的有关flink的问题及回答。
0
文章
377
问答
问答排行榜
最热
最新
推荐问答
相关电子书
更多
文娱运维技术
立即下载
《SaaS模式云原生数据仓库应用场景实践》
立即下载
《看见新力量:二》电子书
立即下载