如图
使用FileSystem按字节数读取文件时出现了一行数据不全的情况。代码如下：//text格式存储
Path filePath = new Path(location);
FileStatus[] status = null;
FileSystem fs = null;
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://192.168.20.181:8020");
conf.set("fs.permissions.umask-mode", "002");
System.getProperties().setProperty("HADOOP_USER_NAME", "hdfs");
try {
fs = FileSystem.get(conf);
} catch (Exception e) {
e.printStackTrace();
logger.error("获取hdfs目录列表失败！");
}
try {
status = fs.listStatus(filePath);
} catch (Exception e) {
e.printStackTrace();
throw new Exception("HdfsListDirectory:" + e.getMessage());
}
long start = (new Date()).getTime();
List<Future> taskList = new ArrayList<Future>();
for (FileStatus st : status) {
MyCallable myCallable = new MyCallable(st, fs,lineDelim,fieldDelim,cloumnList);
FutureTask task = new FutureTask(myCallable);
pool.submit(task);
taskList.add(task);
}
int count = 0;
for(Future f:taskList){
count =count +(Integer)f.get();
}
请问各位大佬：java有什么方法读取hive中textfile文件，orc文件的读取已经完成。
版权声明：本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
自问自答一次。可以使用LazySimpleSerDe去读取文件，思路来自读取orc文件。具体代码如下：
//text格式存储
Path filePath = new Path(location);
Properties p = new Properties();
LazySimpleSerDe serde = new LazySimpleSerDe();
JobConf conf = new JobConf();
conf.set("fs.default.name","hdfs://192.168.20.181:8020");
p.setProperty("columns", cloNameSB.toString());
p.setProperty("columns.types", cloTypeSB.toString());
serde.initialize(conf, p);
StructObjectInspector inspector = (StructObjectInspector) serde.getObjectInspector();
TextInputFormat in = new TextInputFormat();
in.configure(conf);
TextInputFormat.setInputPaths(conf, filePath);
InputSplit[] splits = in.getSplits(conf, 1);
long start = (new Date()).getTime();
int count = 0;
for(InputSplit split:splits){
RecordReader reader = in.getRecordReader(split, conf, Reporter.NULL);
Object key = reader.createKey();
Object value = reader.createValue();
List<? extends StructField> fields = inspector.getAllStructFieldRefs();
while(reader.next(key, value)) {
Map<String,Object> map = new HashMap<String,Object>();
for(int i = 0; i<fields.size(); i++){
Text text = (Text)value;
String[] vs = text.toString().split(fieldDelim);
map.put(fields.get(i).getFieldName(), vs[i]);
}
count++;
}
}
System.out.println((new Date()).getTime()-start);
System.out.println(count);
这样虽然比直接按字节读取文件要慢一点，但是更准确
