今天收到报警Datanode is dead，登录上去看下发现datanode进程还“活着”，没有高负载，内存也正常，datanode日志发现只有几处block传输异常，之后就是在接收block，但是心跳超时导致被NN认为死亡：
|
1
2
3
4
5
6
7
8
9
|
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException inBlockReceiver.run():
java.net.SocketException:Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:
96
)
at java.net.SocketOutputStream.write(SocketOutputStream.java:
136
)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:
65
)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:
123
)
at java.io.DataOutputStream.flush(DataOutputStream.java:
106
)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:
1003
)
at java.lang.Thread.run(Thread.java:
662
)
|
查看了ThreadDump（见附1）后终于发现原因：
在pipeline失败后，需要创建一个新的DataXceiver进行pipeline recovery,1.调用recoverRbw方法会锁定FsDatasetImp；2.然后调用stopWriter:即在进行错误恢复时，如果该数据节点的写线程DataReceiver如果还没有结束，要首先中断并等待该线程退出，保证后续恢复：
|
1
2
3
4
5
6
7
8
9
10
|
public
void
stopWriter()
throws
IOException {
if
(writer !=
null
&& writer != Thread.currentThread() && writer.isAlive()) {
writer.interrupt();
try
{
writer.join();
}
catch
(InterruptedException e) {
throw
new
IOException(
"Waiting for writer thread is interrupted."
);
}
}
}
|
3.同时老的写入线程可能正在提交block，因此需要对FsDatasetImpl进行锁定，这时死锁发生了！！！
4.而Datanode发送Heartbeat前也需要获取FsDatasetImpl的锁定获取节点容量，因此发生超时！
其实这是一个Bug，在CDH4.4.0已经修复了，引入隐藏参数dfs.datanode.xceiver.stop.timeout.millis默认1min，对stopWriter引入超时join.
附1：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
"DataXceiverfor client xxx at xxx [Receiving block xxx]"
daemon prio=10tid=
0x00000000406bc000
nid=
0x612d
in Object.wait() [
0x00007ffdcb314000
]
java.lang.Thread.State: WAITING (on objectmonitor)
at java.lang.Object.wait(Native Method)
atjava.lang.Thread.join(Thread.java:
1186
)
- locked <
0x00000007ca882a30
> (aorg.apache.hadoop.util.Daemon)
atjava.lang.Thread.join(Thread.java:
1239
)
atorg.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.stopWriter(ReplicaInPipeline.java:
157
)
atorg.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverRbw(FsDatasetImpl.java:
707
)
- locked <
0x0000000784acf478
> (aorg.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
atorg.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverRbw(FsDatasetImpl.java:
91
)
atorg.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:
163
)
atorg.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:
457
)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:
103
)
atorg.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:
67
)
atorg.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
221
)
at java.lang.Thread.run(Thread.java:
662
)
|
|
1
2
3
4
5
6
|
"PacketResponder:xxx, type=HAS_DOWNSTREAM_IN_PIPELINE"
daemon prio=10tid=
0x00007ffde0248000
nid=
0x5ed7
waiting
for
monitor entry [
0x00007ffdccbf9000
]
java.lang.Thread.State: BLOCKED (on objectmonitor)
atorg.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeBlock(FsDatasetImpl.java:
846
)
- waiting to lock<
0x0000000784acf478
> (a org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
atorg.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:
964
)
atjava.lang.Thread.run(Thread.java:
662
)
|
本文转自MIKE老毕 51CTO博客，原文链接：http://blog.51cto.com/boylook/1313655，如需转载请自行联系原作者
