[解决]java.io.IOException: Cannot obtain block length for LocatedBlock

简介: 在hadoop测试集群运行job的过程中发现部分运行失败,有Cannot obtain block length for LocatedBlock,使用hdfs dfs -cat ${文件}的时候也报这个错,看过代码后发现是文件未被正常关闭(flume传到hdfs上的,可能由于Namenode长时间gc或者切换或者重启会留下很多.

在hadoop测试集群运行job的过程中发现部分运行失败,有Cannot obtain block length for LocatedBlock,使用hdfs dfs -cat ${文件}的时候也报这个错,看过代码后发现是文件未被正常关闭(flume传到hdfs上的,可能由于Namenode长时间gc或者切换或者重启会留下很多.tmp结尾的文件),为此,使用hadoop命令显示处于打开状态的文件,然后删除这些文件:

hadoop fsck / -openforwrite | egrep -v '^\.+$' | egrep "MISSING|OPENFORWRITE" | grep -o "/[^ ]*" | sed -e "s/:$//" | xargs -i hadoop fs -rmr {};

然后重传这些文件到hdfs。

目录
相关文章
|
2月前
启动报错:java.nio.charset.MalformedInputException: Input length = 1
启动报错:java.nio.charset.MalformedInputException: Input length = 1
18 0
|
7月前
|
Java Spring
【Java异常】java.lang.ClassCastException: java.io.NotSerializableException cannot be cast to java.lang.S
【Java异常】java.lang.ClassCastException: java.io.NotSerializableException cannot be cast to java.lang.S
34 0
java.lang.Error: Unresolved compilation problem: The type List is not generic; it cannot be parame
java.lang.Error: Unresolved compilation problem: The type List is not generic; it cannot be parame
|
4月前
|
XML Java Maven
nested exception is java.io.FileNotFoundException: class path resource [springmvc.xml] cannot be ope
nested exception is java.io.FileNotFoundException: class path resource [springmvc.xml] cannot be ope
76 0
nested exception is java.io.FileNotFoundException: class path resource [springmvc.xml] cannot be ope
|
1月前
Caused by: java.lang.IllegalStateException: Ambiguous mapping. Cannot map ‘quanZiController‘ method
Caused by: java.lang.IllegalStateException: Ambiguous mapping. Cannot map ‘quanZiController‘ method
14 0
|
2月前
|
网络安全
ssh报错java.lang.ClassCastException: com.sun.proxy.$Proxy6 cannot be cast to org.service.impl.EmpServi
ssh报错java.lang.ClassCastException: com.sun.proxy.$Proxy6 cannot be cast to org.service.impl.EmpServi
11 1
|
2月前
【问题篇】java.nio.charset.MalformedInputException: Input length = 1解决
【问题篇】java.nio.charset.MalformedInputException: Input length = 1解决
27 0
|
2月前
|
Java Spring
解决org.yaml.snakeyaml.error.YAMLException: java.nio.charset.MalformedInputException: Input length = 2
解决org.yaml.snakeyaml.error.YAMLException: java.nio.charset.MalformedInputException: Input length = 2
39 1
|
7月前
|
Java 关系型数据库 MySQL
【Java异常】java.sql.SQLExcetion:Cannot convert value “0000-00-00 00:00:00” from column 9 to TIMESTAMP
【Java异常】java.sql.SQLExcetion:Cannot convert value “0000-00-00 00:00:00” from column 9 to TIMESTAMP
45 0
|
5月前
|
Java
【Java报错】java.lang.ClassCastException: xxxClass cannot be cast to java.lang.Comparable 问题重现+解决
【Java报错】java.lang.ClassCastException: xxxClass cannot be cast to java.lang.Comparable 问题重现+解决
50 0