今天有个etl开发在drop partition的时候遇到了问题,因为是使用了自己的账号,而hdfs中对应partition的文件属主是hdfs的,在删除时会因为权限问题报错,切换用户为hdfs,做drop partition的错误,还是报错,看来没这么简单。
查看表的hdfs属性,目录的属主不是hdfs且目录对 hdfs没有写权限:
1
2
3
|
[hdfs
@nfzm
~]$ hadoop fs -ls -d hdfs:
//xxxx:9000/bip/external_table/vipdw/droptest
Found
1
items
drwxr-x r-x - ericni hdfs
0
2014
-
08
-
21
17
:
13
hdfs:
//xxxxx:9000/bip/external_table/vipdw/droptest
|
drop partition报错,报hdfs用户没有权限,在hive中是设置了hdfs为超级管理员的,所以如果直接用hadoop fs命令是可以删除的,也就是权限问题不是出在hdfs上,而是在hive上,这里怎么会没有权限呢?
1
2
|
hive (vipdw)> ALTER TABLE droptest DROP PARTITION(dt=
'20140840'
);
14
/
08
/
21
17
:
31
:
21
ERROR metastore.RetryingHMSHandler: MetaException(message:Table partition not deleted since hdfs:
//xxxx:9000/bip/external_table/vipdw/droptest is not writable by hdfs
|
根据调用信息,异常是在HiveMetaStore类中的verifyIsWritablePath 方法中抛出, verifyIsWritablePath类用来判断本目录的上层目录对当前用户是否可写,按照一般的思路,所有的hdfs目录都应该对超级管理员可写的。
1
2
3
4
5
6
7
8
9
10
11
12
|
private
void
verifyIsWritablePath(Path dir)
throws
MetaException {
try
{
if
(!wh.isWritable(dir.getParent())) {
throw
new
MetaException(
"Table partition not deleted since "
+ dir.getParent()
+
" is not writable by "
+ hiveConf.getUser());
}
}
catch
(IOException ex) {
LOG.warn(
"Error from isWritable"
, ex);
throw
new
MetaException(
"Table partition not deleted since "
+ dir.getParent()
+
" access cannot be checked: "
+ ex.getMessage());
}
}
|
这里判断调用了 Warehouse类的isWritable 的方法,但是实现方法有些问题,不会区分超级用户,只会根据目录的属主和属组的权限做判断。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
|
public
boolean
isWritable(Path path)
throws
IOException {
if
(!storageAuthCheck) {
// no checks for non-secure hadoop installations
return
true
;
}
if
(path ==
null
) {
//what??!!
return
false
;
}
final
FileStatus stat;
try
{
stat = getFs(path).getFileStatus(path);
}
catch
(FileNotFoundException fnfe){
// File named by path doesn't exist; nothing to validate.
return
true
;
}
catch
(Exception e) {
// all other exceptions are considered as emanating from
// unauthorized accesses
return
false
;
}
final
UserGroupInformation ugi;
try
{
ugi = ShimLoader.getHadoopShims().getUGIForConf(conf);
}
catch
(LoginException le) {
throw
new
IOException(le);
}
String user = ShimLoader.getHadoopShims().getShortUserName(ugi);
//check whether owner can delete
if
(stat.getOwner().equals(user) &&
stat.getPermission().getUserAction().implies(FsAction.WRITE)) {
// 判断属主有没有写权限
return
true
;
}
//check whether group of the user can delete
if
(stat.getPermission().getGroupAction().implies(FsAction.WRITE)) {
// 判断属组有没有有写权限
String[] groups = ugi.getGroupNames();
if
(ArrayUtils.contains(groups, stat.getGroup())) {
return
true
;
}
}
//check whether others can delete (uncommon case!!) // // 判断other有没有写权限
if
(stat.getPermission().getOtherAction().implies(FsAction.WRITE)) {
return
true
;
}
return
false
;
}
|
fix这个bug也比较简单:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
git diff
diff --git a/metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java b/metastore/src/java/org/apache/hadoop/hive/metastor
index a4f9b0a..c69e49d
100755
--- a/metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java
+++ b/metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java
@@ -
263
,
7
+
263
,
17
@@
public
boolean
isWritable(Path path)
throws
IOException {
}
String user = ShimLoader.getHadoopShims().getShortUserName(ugi);
//check whether owner can delete
-
if
(stat.getOwner().equals(user) &&
+
+
//fix hdfs drop partition bug
+ LOG.warn(
"in Warehouse user is "
+ user);
+ String adminStr = HiveConf.getVar(conf,HiveConf.ConfVars.USERS_IN_ADMIN_ROLE,
"hdfs"
).trim();
+ LOG.warn(
"adminStr is "
+ adminStr);
+
if
(user.equals(adminStr)){
+
return
true
;
+ }
+
+
+
if
(stat.getOwner().equals(user) &&
stat.getPermission().getUserAction().implies(FsAction.WRITE)) {
return
true
;
}
|
本文转自菜菜光 51CTO博客,原文链接:http://blog.51cto.com/caiguangguang/1543652,如需转载请自行联系原作者