一、问题现象
某一个nodemanager退出后，导致 application master中出现大量的如下日志，并且持续很长时间，application master才成功退出。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
2016-06-24 09:32:35,596 INFO [ContainerLauncher
#3] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 09:32:35,596 INFO [ContainerLauncher
#9] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 09:32:35,597 INFO [ContainerLauncher
#7] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 09:32:36,455 INFO [ContainerLauncher
#8] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 09:32:36,539 INFO [ContainerLauncher
#5] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 09:32:36,539 INFO [ContainerLauncher
#1] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 09:32:36,539 INFO [ContainerLauncher
#6] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 09:32:36,539 INFO [ContainerLauncher
#2] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 09:32:36,539 INFO [ContainerLauncher
#0] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 09:32:36,596 INFO [ContainerLauncher
#4] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 09:32:36,597 INFO [ContainerLauncher
#3] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 09:32:36,597 INFO [ContainerLauncher
#9] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
.......
2016-06-24 12:57:52,328 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206
/192
.168.1.199:32951. Already tried 8
time
(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 12:57:53,339 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206
/192
.168.1.199:32951. Already tried 9
time
(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 12:58:04,357 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206
/192
.168.1.199:32951. Already tried 0
time
(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 12:58:05,367 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206
/192
.168.1.199:32951. Already tried 1
time
(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 12:58:06,378 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206
/192
.168.1.199:32951. Already tried 2
time
(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 12:58:07,392 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206
/192
.168.1.199:32951. Already tried 3
time
(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 12:58:08,399 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206
/192
.168.1.199:32951. Already tried 4
time
(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 12:58:09,408 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206
/192
.168.1.199:32951. Already tried 5
time
(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 12:58:10,417 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206
/192
.168.1.199:32951. Already tried 6
time
(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 12:58:11,425 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206
/192
.168.1.199:32951. Already tried 7
time
(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-24 12:58:12,434 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206
/192
.168.1.199:32951. Already tried 8
time
(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
|
1）dchadoop206上的nodemanager退出后（由于重启），导致application master持续的去连接之前nodemanager上的container。显然这些container是已经连接不上了。
2）最终经过非常长的时间大概3-4小时后，连接不上的异常才抛出，application master正常结束。
|
1
|
<strong style=
"font-family:'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif;white-space:normal;color:rgb(51,51,51);font-size:1.4em;line-height:1.1;"
><span style=
"font-family:'黑体', SimHei;font-size:16px;"
>二、问题分析<
/span
><
/strong
><br>
|
这个问题主要涉及hadoop的rpc机制。首先看下面两个配置参数
|
1
2
3
4
5
6
7
8
9
10
|
#定义client连接到nodemanager的最大超时时间,不是单次连接，而是经过多少时间连接不上nodemanager，则认为操作失败
<property>
<name>yarn.client.nodemanager-connect.max-wait-ms<
/name
>
<value>15*60*1000<
/value
>
<
/property
>
# 定义每次尝试去连接nodemanager的时间间隔
<property>
<name>yarn.client.nodemanager-connect.retry-interval-ms<
/name
>
<value>10*1000<
/value
>
<
/property
>
|
根据这两个参数的定义，ApplicationMaster经过15分钟仍然连不上nodemanager的container，会取消try connect。但观察的情况是Application Master 需要等大约30分钟，才取消try connect。主要原因在于hadoop 的rpc机制如下，首先ApplicationMaster 会根据上面的两个参数，构造一个RetryUpToMaximumCountWithFixedSleep的重连策略，这个重连策略会通过以下方式计算
MaximumCount：yarn.client.nodemanager-connect.max-wait-ms/yarn.client.nodemanager-connect.retry-interval-ms=90次
而每次的RPC请求中，Client也有自己的重连策略，就是类似这样的东东:
|
1
2
|
2016-06-24 09:32:36,455 INFO [ContainerLauncher
#8] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
由两个rpc参数控制，ipc.client.connect.max.retries=10和ipc.client.connect.retry.interval=1000ms控制
|
所以最终ApplicationMaster 放弃try connect的等待时间是：90*(10+10)=1800s
三、解决办法
1）在提交map-reduce/hive sql/hive server2等客户端机器修改yarn-site.xml的以下参数
2）hadoop命令行中通过-D设置该参数
|
1
2
3
4
|
<property>
<name>yarn.client.nodemanager-connect.max-wait-ms<
/name
>
<value>180000<
/value
>
<
/property
>
|
这样总的等待时间就是6分钟。
注意事项
这个修改是不需要做任何重启yarn组件操作的，是一个客户端相关的操作！
本文转自 zouqingyun 51CTO博客，原文链接：http://blog.51cto.com/zouqingyun/1881294，如需转载请自行联系原作者
