flink Application Native k8s使用oss作为backend日志偶尔报错怎么

环境：Native Kubernetes

模式：Application Mode

描述：

Flink以Native Kubernetes Application模式运行在k8s时，使用filesystem OSS作为backend发现日志请求OSS报错。

当代码使用`source.setStartFromEarliest();`，启动job之后从头开始消费，运行过程正常，运行到最新点位后会出现以下报错，过一段时间或者重启job之后报错消失。

当代码使用`source.setStartFromLatest();`，启动job之后直接从最新点位开始消费，则不会出现此报错。

据观察请问是我哪里配置或者使用有问题么？

命令：

./bin/flink run-application \

--target kubernetes-application \

-Dkubernetes.cluster-id=demo \

-Dkubernetes.container.image=xx/xx/xx:2.0.16 \

-Dstate.backend=filesystem \

-Dstate.checkpoints.dir=oss://bucket/文件夹 \

-Dfs.oss.endpoint=oss-cn-beijing-internal.aliyuncs.com<http://oss-cn-beijing-internal.aliyuncs.com> \

-Dfs.oss.accessKeyId=xx \

-Dfs.oss.accessKeySecret=xx \

local:///opt/flink/usrlib/my-flink-job.jar



报错日志：



2021-03-03 02:53:46,133 INFO org.apache.flink.streaming.connectors.pulsar.internal.PulsarMetadataReader [] - Committing offset 12701:1:-1:4 to topic TopicRange{topic=persistent://public/xx/xxxx, key-range=SerializableRange{range=[0, 65535]}}

2021/3/3 上午10:53:46 2021-03-03 02:53:46,140 INFO org.apache.flink.streaming.connectors.pulsar.internal.PulsarMetadataReader [] - Successfully committed offset 12701:1:-1:4 to topic TopicRange{topic=persistent://public/xx/xxxx, key-range=SerializableRange{range=[0, 65535]}}

2021/3/3 上午10:53:50 2021-03-03 02:53:50,899 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found

2021/3/3 上午10:53:50 [ErrorCode]: NoSuchKey

2021/3/3 上午10:53:50 [RequestId]: xx

2021/3/3 上午10:53:50 [HostId]: null

2021/3/3 上午10:53:50 2021-03-03 02:53:50,904 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found

2021/3/3 上午10:53:50 [ErrorCode]: NoSuchKey

2021/3/3 上午10:53:50 [RequestId]: xx

2021/3/3 上午10:53:50 [HostId]: null



kill进程pod重启或过一段时间后taskManager正常日志：



2021-03-03 03:18:21,602 INFO org.apache.flink.streaming.connectors.pulsar.internal.PulsarMetadataReader [] - Successfully committed offset 12716:7:-1:1 to topic TopicRange{topic=persistent://public/xx/xxxx, key-range=SerializableRange{range=[0, 65535]}}

2021/3/3 上午11:18:26 2021-03-03 03:18:26,573 INFO org.apache.flink.streaming.connectors.pulsar.internal.PulsarMetadataReader [] - Committing offset 12716:7:-1:1 to topic TopicRange{topic=persistent://public/xx/xxxx, key-range=SerializableRange{range=[0, 65535]}}

2021/3/3 上午11:18:26 2021-03-03 03:18:26,582 INFO org.apache.flink.streaming.connectors.pulsar.internal.PulsarMetadataReader [] - Successfully committed offset 12716:7:-1:1 to topic TopicRange{topic=persistent://public/xx/xxxx, key-range=SerializableRange{range=[0, 65535]}}

2021/3/3 上午11:18:31 2021-03-03 03:18:31,571 INFO org.apache.flink.streaming.connectors.pulsar.internal.PulsarMetadataReader [] - Committing offset 12716:7:-1:1 to topic TopicRange{topic=persistent://public/xx/xxxx, key-range=SerializableRange{range=[0, 65535]}}

2021/3/3 上午11:18:31 2021-03-03 03:18:31,580 INFO org.apache.flink.streaming.connectors.pulsar.internal.PulsarMetadataReader [] - Successfully committed offset 12716:7:-1:1 to topic TopicRange{topic=persistent://public/xx/xxxx, key-range=SerializableRange{range=[0, 65535]}}

2021/3/3 上午11:18:36 2021-03-03 03:18:36,633 INFO org.apache.flink.streaming.connectors.pulsar.internal.PulsarMetadataReader [] - Committing offset 12716:7:-1:1 to topic TopicRange{topic=persistent://public/xx/xxxx, key-range=SerializableRange{range=[0, 65535]}}

2021/3/3 上午11:18:36 2021-03-03 03:18:36,642 INFO org.apache.flink.streaming.connectors.pulsar.internal.PulsarMetadataReader [] - Successfully committed offset 12716:7:-1:1 to topic TopicRange{topic=persistent://public/xx/xxxx, key-range=SerializableRange{range=[0, 65535]}}



oss内文件：

[cid:AD54B42B-CDD0-4EAC-93A8-BCD11C8D1904]

chk-10880目录：

[cid:FD31EF7D-6D35-4B26-9A2A-FA13170EBB70]*来自志愿者整理的flink邮件归档