Flink 1.5.4没有在Kubernetes中注册Google云端存储(GCS)文件系统,尽管它在docker容器中有效-问答-阿里云开发者社区-阿里云

开发者社区> 问答> 正文
阿里云
为了无法计算的价值
打开APP
阿里云APP内打开

Flink 1.5.4没有在Kubernetes中注册Google云端存储(GCS)文件系统,尽管它在docker容器中有效

2018-12-10 10:04:02 3033 1

我试图启动一个flink-jobmanager坚持其在GCS状态,所以我加了high-availability.storageDir: gs://BUCKET/ha一行到我flink-conf.yaml和我建立我的Dockerfile描述这里

这是我的Dockerfile:

FROM flink:1.5-hadoop28

ADD https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar /opt/flink/lib/gcs-connector-latest-hadoop2.jar

RUN mkdir /opt/flink/etc-hadoop
COPY flink-conf.yaml /opt/flink/conf/flink-conf.yaml
COPY key.json /opt/flink/etc-hadoop/key.json
COPY core-site.xml /opt/flink/etc-hadoop/core-site.xml
现在,如果我通过构建此容器docker build -t flink:dev .并在其中启动交互式shell docker run -ti flink:dev /bin/bash,我可以通过以下方式启动flink jobmanager:

flink-console.sh jobmanager --configDir=/opt/flink/conf/ --executionMode=cluster

Flink正在开启ar并正常启动。但是,当我使用以下yaml在Kubernetes上启动它时,基于这里的一个:

apiVersion: apps/v1
kind: Deployment
metadata:
name: flink-jobmanager
spec:
replicas: 1
selector:

matchLabels:
  app: flink
  component: jobmanager

template:

metadata:
  labels:
    app: flink
    component: jobmanager
spec:
  containers:
  - name: jobmanager
    image: flink:dev
    imagePullPolicy: Always
    resources:
      requests:
        memory: "1024Mi"
        cpu: "500m"
      limits:
        memory: "2Gi"
        cpu: "1000m"
    args: ["jobmanager"]
    ports:
    - containerPort: 6123
      name: rpc
    - containerPort: 6124
      name: blob
    - containerPort: 6125
      name: query
    - containerPort: 8081
      name: ui
    - containerPort: 46110
      name: ha
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /opt/flink/etc-hadoop/key.json
    - name: JOB_MANAGER_RPC_ADDRESS
      value: flink-jobmanager

Flink似乎无法注册文件系统:

2018-10-04 09:20:51,357 DEBUG org.apache.flink.runtime.util.HadoopUtils - Cannot find hdfs-default configuration-file path in Flink config.
2018-10-04 09:20:51,358 DEBUG org.apache.flink.runtime.util.HadoopUtils - Cannot find hdfs-site configuration-file path in Flink config.
2018-10-04 09:20:51,359 DEBUG org.apache.flink.runtime.util.HadoopUtils - Adding /opt/flink/etc-hadoop//core-site.xml to hadoop configuration
2018-10-04 09:20:51,767 DEBUG org.apache.hadoop.security.UserGroupInformation - PrivilegedActionException as:flink (auth:SIMPLE) cause:java.io.IOException: Could not create FileSystem for highly available storage (high-availability.storageDir)
2018-10-04 09:20:51,767 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Cluster initialization failed.
java.io.IOException: Could not create FileSystem for highly available storage (high-availability.storageDir)

    at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:122)
    at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:95)
    at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:115)
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:402)
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:270)
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:225)
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:189)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
    at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:188)
    at org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint.main(StandaloneSessionClusterEntrypoint.java:91)

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'gs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.

    at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:405)
    at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
    at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
    at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:119)
    ... 12 more

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop File System abstraction does not support scheme 'gs'. Either no file system implementation exists for that scheme, or the relevant classes are missing from the classpath.

    at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:102)
    at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
    ... 15 more

Caused by: java.io.IOException: No FileSystem for scheme: gs

    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
    at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:99)
    ... 16 more

由于Kubernetes应该使用相同的图像,我很困惑这是如何做到的。我需要在这里监督一些事件吗?

取消 提交回答
全部回答(1)
  • flink小助手
    2019-07-17 23:19:02

    问题是使用dev标签。使用特定版本标签修复了问题。

    0 0
相关问答

1

回答

容器中的Docker:是什么?

2021-12-14 20:06:38 777浏览量 回答数 1

1

回答

docker容器是什么呢?

2021-10-25 19:38:29 339浏览量 回答数 1

1

回答

Docker容器是什么?

2021-10-21 14:25:46 242浏览量 回答数 1

1

回答

docker容器中的Xdebug无法连接到Qubes OS上的PhpStorm

2020-01-08 15:02:29 464浏览量 回答数 1

1

回答

Docker容器未与主机mariaDB连接

2020-01-06 15:56:20 315浏览量 回答数 1

1

回答

如何从Docker容器访问JACK音频

2020-01-06 16:00:39 455浏览量 回答数 1

1

回答

Docker-如何将当前容器另存为tar文件并将其传输到另一台计算机?

2020-01-05 19:05:47 484浏览量 回答数 1

1

回答

Docker有哪些关于容器的基本操作?

2019-12-18 10:13:37 544浏览量 回答数 1

1

回答

如何从现有的docker容器创建pod

2019-12-11 21:17:33 287浏览量 回答数 1

1

回答

现在docker容器中跑matlab程序,不想安装matlab, 怎么做呢

2019-12-09 18:08:46 2605浏览量 回答数 1
+关注
flink小助手
flink小助手会定期更新直播回顾等资料和文章干货,还整合了大家在钉群提出的有关flink的问题及回答。
0
文章
377
问答
问答排行榜
最热
最新
相关电子书
更多
低代码开发师(初级)实战教程
立即下载
阿里巴巴DevOps 最佳实践手册
立即下载
冬季实战营第三期:MySQL数据库进阶实战
立即下载