作业因为异常restart后，频繁OOM

目前遇到的问题是作业如果因为网络抖动或者硬件故障导致的pod被失联而fail，在pod重生后，作业自动restart，作业运行一段时间（半小时到1小时不等）很容易出现其他pod因为oom被os

kill的现象，然后反复循环，pod 被kill越来越频繁。目前的解决方法是手动销毁这个集群，重新构建一个集群后重启作业，就恢复正常。

如果单纯heap的状态后台，作业restart不会出现这样的问题。

有一些不成熟的猜测，作业在fail后，native memory没有释放干净，pod的limit假设为10G，那么job

restart后只有8G，TM还是按照10G的标准运行，pod使用的内存就会超过10G而被os kill（纯属猜测）。

请问大家是否有什么好的提议或者解决方法？

其中一次系统内核日志如下：

Jun 30 21:59:15 flink-tm-1 kernel: memory: usage 28672000kB, limit

28672000kB, failcnt 11225

Jun 30 21:59:15 flink-tm-1 kernel: memory+swap: usage 28672000kB, limit

9007199254740988kB, failcnt 0

Jun 30 21:59:15 flink-tm-1 kernel: kmem: usage 0kB, limit

9007199254740988kB, failcnt 0

Jun 30 21:59:15 flink-tm-1 kernel: Memory cgroup stats for

/kubepods.slice/kubepods-pod5ad5d2ea_5faa_4a11_96b4_39271ab76e99.slice:

cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0K

B inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB

unevictable:0KB

Jun 30 21:59:15 flink-tm-1 kernel: Memory cgroup stats for

/kubepods.slice/kubepods-pod5ad5d2ea_5faa_4a11_96b4_39271ab76e99.slice/docker-fe101418a3b2a7c534e89b4ac73d29b04070eb923220a5b1

7338850bbdb3817a.scope: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB

swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB

active_file:0KB unevictable:0KB

Jun 30 21:59:15 flink-tm-1 kernel: Memory cgroup stats for

/kubepods.slice/kubepods-pod5ad5d2ea_5faa_4a11_96b4_39271ab76e99.slice/docker-a2295e812a828738810a8f1ae69cd48e99ef98b9e1038158a6e33f81524cc02a.scope:

cache:180KB rss:28671776KB rss_huge:26437632KB mapped_file:144KB swap:0KB

inactive_anon:0KB active_anon:28671760KB inactive_file:4KB

active_file:4KB

unevictable:0KB

Jun 30 21:59:15 flink-tm-1 kernel: [ pid ] uid tgid total_vm rss

nr_ptes swapents oom_score_adj name

Jun 30 21:59:15 flink-tm-1 kernel: [16875] 0 16875 253 1

4 0 -998 pause

Jun 30 21:59:15 flink-tm-1 kernel: [17274] 0 17274 1369 421

7 0 -998 bash

Jun 30 21:59:15 flink-tm-1 kernel: [18089] 0 18089 10824832 7174316

14500 0 -998 java

Jun 30 21:59:15 flink-tm-1 kernel: [18348] 0 18348 1017 196

6 0 -998 tail

Jun 30 21:59:15 flink-tm-1 kernel: Memory cgroup out of memory: Kill

process 26824 (Window(Tumbling) score 4 or sacrifice child

Jun 30 21:59:15 flink-tm-1 kernel: Killed process 18089 (java)

total-vm:43299328kB, anon-rss:28669084kB, file-rss:28180kB, shmem-rss:0kB

Looking forward to your reply and help.*来自志愿者整理的flink邮件归档