当我flink on yarn 运行的flink程序 突然遇到hdfs的服务出现了异常,导致ark1的hdfs服务从active变成了standby
ark2进行了接手从standby变为active
问题:我flink程序设置的checkpoint的hdfs的url是hdfs:ark:8082 现在就出现了异常,导致无法读取standby节点的快照数据,一直失败
我现在设置了hdfs的mycluster 但是checkpoint是读不到这个路径
final String HADOOP_CONF_DIR = "/etc/hadoop/conf";
org.apache.hadoop.conf.Configuration configuration = new org.apache.hadoop.conf.Configuration();
configuration.addResource(new Path(HADOOP_CONF_DIR + "/core-site.xml"));
configuration.addResource(new Path(HADOOP_CONF_DIR + "/hdfs-site.xml"));
env.setStateBackend(new FsStateBackend("hdfs://mycluster/flinkCheckpoint")); //失败的
报错内容:
Caused by: java.io.IOException: Cannot instantiate file system for URI: hdfs://mycluster/flinkCheckpoint
at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:196)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:527)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:408)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:64)
at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:518)
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:321)
... 22 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:438)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:690)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:631)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:160)
at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:168)
... 28 common frames omitted
Caused by: java.net.UnknownHostException: mycluster
... 35 common frames omitted
*来自志愿者整理的flink邮件归档
UnknownHostException: mycluster 请检查下环境配置或集群状态*来自志愿者整理的flink邮件归档
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。