开发者社区> 问答> 正文

checkpoint的路径问题

当我flink on yarn 运行的flink程序 突然遇到hdfs的服务出现了异常,导致ark1的hdfs服务从active变成了standby
ark2进行了接手从standby变为active
问题:我flink程序设置的checkpoint的hdfs的url是hdfs:ark:8082  现在就出现了异常,导致无法读取standby节点的快照数据,一直失败


我现在设置了hdfs的mycluster 但是checkpoint是读不到这个路径
final String HADOOP_CONF_DIR = "/etc/hadoop/conf";
org.apache.hadoop.conf.Configuration configuration = new org.apache.hadoop.conf.Configuration();
configuration.addResource(new Path(HADOOP_CONF_DIR + "/core-site.xml"));
configuration.addResource(new Path(HADOOP_CONF_DIR + "/hdfs-site.xml"));

env.setStateBackend(new FsStateBackend("hdfs://mycluster/flinkCheckpoint")); //失败的

报错内容:
Caused by: java.io.IOException: Cannot instantiate file system for URI: hdfs://mycluster/flinkCheckpoint
at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:196)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:527)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:408)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:64)
at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:518)
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:321)
... 22 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:438)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:690)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:631)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:160)
at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:168)
... 28 common frames omitted
Caused by: java.net.UnknownHostException: mycluster
... 35 common frames omitted
*来自志愿者整理的flink邮件归档

展开
收起
moonlightdisco 2021-12-08 09:47:21 526 0
1 条回答
写回答
取消 提交回答
  • UnknownHostException: mycluster 请检查下环境配置或集群状态*来自志愿者整理的flink邮件归档

    2021-12-08 10:05:18
    赞同 展开评论 打赏
问答分类:
问答地址:
问答排行榜
最热
最新

相关电子书

更多
俞航翔|基于Log的通用增量Checkpoint 立即下载
低代码开发师(初级)实战教程 立即下载
阿里巴巴DevOps 最佳实践手册 立即下载