因为某种历史原因集群出现了一批corrupted文件。读取这些文件会报“BlockMissingException”异常,例如:
6/11/08 19:04:20 WARN hdfs.DFSClient: DFS Read
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-221196964-172.23.64.95-1477965231106:blk_1073748785_7961 file=/hadoop-2.7.3.tar.gz
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:983)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
at java.io.DataInputStream.read(DataInputStream.java:100)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:91)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:466)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:391)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:328)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:263)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:248)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:243)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:220)
at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
AI 代码解读
而通过fsck命令可以看到,corrupted文件只丢失了一个或几个block,还有多个block残留,例如:
bin/hdfs fsck /hadoop-2.7.3.tar.gz -files -blocks
Connecting to namenode via http://localhost:50070/fsck?ugi=user&files=1&blocks=1&path=%2Fhadoop-2.7.3.tar.gz
FSCK started by user (auth:SIMPLE) from /127.0.0.1 for path /hadoop-2.7.3.tar.gz at Tue Nov 08 19:03:38 CST 2016
/hadoop-2.7.3.tar.gz 214092195 bytes, 2 block(s): Under replicated BP-221196964-172.23.64.95-1477965231106:blk_1073748784_7960. Target Replicas is 3 but found 1 replica(s).
/hadoop-2.7.3.tar.gz: CORRUPT blockpool BP-221196964-172.23.64.95-1477965231106 block blk_1073748785
MISSING 1 blocks of total size 79874467 B
0. BP-221196964-172.23.64.95-1477965231106:blk_1073748784_7960 len=134217728 repl=1
1. BP-221196964-172.23.64.95-1477965231106:blk_1073748785_7961 len=79874467 MISSING!
AI 代码解读
在Missing Block找回无望情况下,尽可能降低损失的办法是恢复残留的数据。但是读取corrupted文件又会失败,怎么处理呢?答案就是hack代码。需要hack的类是DFSInputStream,原理就是遇到Missing Block就跳过,能读多少读多少。以下是hack的diff:
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
@@ -882,7 +882,20 @@ private synchronized int readWithStrategy(ReaderStrategy strategy, int off, int
// currentNode can be left as null if previous read had a checksum
// error on the same block. See HDFS-3067
if (pos > blockEnd || currentNode == null) {
- currentNode = blockSeekTo(pos);
+ while (true) {
+ try {
+ currentNode = blockSeekTo(pos);
+ } catch (BlockMissingException e) {
+ LocatedBlock targetBlock = getBlockAt(pos);
+ DFSClient.LOG.warn("Ignore BlockMissingException, try next block " + targetBlock.getBlock());
+ pos += targetBlock.getBlockSize();
+ if (pos >= getFileLength()) {
+ return -1;
+ }
+ continue;
+ }
+ break;
+ }
}
int realLen = (int) Math.min(len, (blockEnd - pos + 1L));
synchronized(infoLock) {
AI 代码解读
hack之后替换下客户端的hadoop-hdfs-2.7.2.jar即可。这样就可以下载文件,或者拷贝到HDFS其他路径。由于遇到读取失败的Block,DFSClient会重试三次,中间会sleep一段时间。为了加快速度可以把sleep时间缩短,比如设置dfs.client.retry.window.base=1。
以上为一种恢复corrupted文件残留数据的一种方式。当然HDFS数据管理上策是设置3个备份,中策是一旦发现corrupted文件,尽量到磁盘上找回block数据。下策才是本文介绍的方法。