最近线上的hive升级到了0.13，遇到不少问题。权限上面，设置了hive.security.authorization.createtable.owner.grants 在hive0.13中，用户自己创建的表也没有权限。通过对源码的分析和debug找到了rc并fix，下面记录下。
1.首先在hive0.11中和hive0.13中分别做建表测试，通过查看数据库中的元数据，发现在hive0.11中如果设置了owner的参数在表创建之后，用户会有这个表的all的权限（具体可以分析db_privs ，tbl_privs表，hive.security.authorization.createtable.owner.grants设置为了all），而在hive0.13中则为空的。
2.通过
|
1
|
hive -hiveconf hive.root.logger=DEBUG,console
|
对比前后的日志,发现在hive0.11的时候，创建的表的属性中有权限的设置
而在hive0.13中这个是为空的
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
|
hive0.
11
:
14
/
07
/
16
17
:
05
:
39
DEBUG hive.log: DDL: struct ericni4 { string a}
14
/
07
/
16
17
:
05
:
39
DEBUG
lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with:
columnNames=[a] columnTypes=[string] separator=[[B
@5b224686
] nullstring=\N
lastColumnTakesRest=false14/
07
/
16
17
:
05
:
39
INFO metastore.HiveMetaStore:
0
:
create_table: Table(tableName:ericni4, dbName:temp, owner:ericni2, createTime:
1405501539
,
lastAccessTime:
0
, retention:
0
,sd:StorageDescriptor(cols:[FieldSchema(name:a,
type:string, comment:
null
)], location:
null
,
inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
compressed:
false
, numBuckets:-
1
,
serdeInfo:SerDeInfo(name:
null
, serializationLib:org.apache.hadoop.hive.serde2.lazy.
LazySimpleSerDe,parameters:{serialization.format=
1
}), bucketCols:[], sortCols:[],
parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
skewedColValueLocationMaps:{}), storedAsSubDirectories:
false
),partitionKeys:[],
parameters:{}, viewOriginalText:
null
, viewExpandedText:
null
, tableType:MANAGED_TABLE,
privileges:PrincipalPrivilegeSet(userPrivileges:{ericni2=[PrivilegeGrantInfo(privilege:
all, createTime:-
1
, grantor:ericni2, grantorType:USER, grantOption:
true
)]},
groupPrivileges:
null
, rolePrivileges:
null
)) #hive0.
11
创建表的时候增加了这个设置
hive0.
13
14
/
07
/
16
17
:
10
:
07
DEBUG hive.log: DDL: struct ericni4 { string a}
14
/
07
/
16
17
:
10
:
07
DEBUG
lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with:
columnNames=[a] columnTypes=[string] separator=[[B
@1667e92a
] nullstring=\N
lastColumnTakesRest=false14/
07
/
16
17
:
10
:
07
INFO metastore.HiveMetaStore:
0
:
create_table: Table(tableName:ericni4, dbName:temp, owner:ericni1, createTime:
1405501807
,
lastAccessTime:
0
, retention:
0
,sd:StorageDescriptor(cols:[FieldSchema(name:a,
type:string, comment:
null
)], location:
null
,
inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
compressed:
false
, numBuckets:-
1
,
serdeInfo:SerDeInfo(name:
null
, serializationLib:org.apache.hadoop.hive.serde2.lazy
.LazySimpleSerDe,parameters:{serialization.format=
1
}), bucketCols:[], sortCols:[],
parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
skewedColValueLocationMaps:{}), storedAsSubDirectories:
false
),partitionKeys:[],
parameters:{}, viewOriginalText:
null
, viewExpandedText:
null
, tableType:MANAGED_TABLE)
|
3.会不会是参数没有生效？在hive0.13中注释掉这个设置做对比，发现create_table的属性并没有改变，说明至少在现在的0.13环境下这个参数并没有生效。而在0.11中前后是有变化的。
4.在hive0.11中我们设置一个错误的值来获取堆栈的调用信息：
得到0.11的调用栈：
|
1
2
3
4
5
6
7
8
9
10
11
12
|
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.checkPrivilege(CreateTableAutomaticGrant.java:
118
)
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.getGrantorInfoList(CreateTableAutomaticGrant.java:
97
)
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.create(CreateTableAutomaticGrant.java:
52
)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:
275
)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:
278
)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:
670
)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:
614
)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39
)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25
)
at java.lang.reflect.Method.invoke(Method.java:
597
)
at org.apache.hadoop.util.RunJar.main(RunJar.java:
208
)
|
再来分析源码
在hive0.11中，在初始化session时，调用SessionState的start方法，会直接根据CreateTableAutomaticGrant类设置createTableGrants属性。
|
1
2
3
4
5
6
7
8
9
10
11
12
|
SessionState.start:
try
{
startSs. authenticator = HiveUtils.getAuthenticator(
startSs.getConf(),HiveConf.ConfVars.HIVE_AUTHENTICATOR_MANAGER);
startSs. authorizer = HiveUtils.getAuthorizeProviderManager(
startSs.getConf(), HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER,
startSs. authenticator);
startSs. createTableGrants = CreateTableAutomaticGrant.create(startSs
.getConf());
//设置owner的相关权限
}
catch
(HiveException e) {
throw
new
RuntimeException(e);
}
|
然后在创建表的时候，调用Hive.createTable的方法
|
1
|
(
public
void
createTable(Table tbl,
boolean
ifNotExists))
|
|
1
2
3
|
public
CreateTableAutomaticGrant getCreateTableGrants() {
return
createTableGrants;
}
|
而在hive0.13中，即使设置错误也不会有报错产生，通过debug源码发现这里的调用关系有些变化，SessionState的getCreateTableGrants方法调用的是setupAuth方法。
|
1
2
3
4
|
public
CreateTableAutomaticGrant getCreateTableGrants() {
setupAuth();
//调用setupAuth方法设置createTableGrants
return
createTableGrants;
}
|
其中setupAuth方法如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
private
void
setupAuth() {
if
(authenticator !=
null
) {
// auth has been initialized
return
;
}
try
{
authenticator = HiveUtils.getAuthenticator( conf,
HiveConf.ConfVars.HIVE_AUTHENTICATOR_MANAGER);
authenticator.setSessionState(
this
);
authorizer = HiveUtils.getAuthorizeProviderManager( conf,
HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER, authenticator,
true
);
if
(authorizer ==
null
) {
//在authorizer 的取值为null时，才会生成权限
// if it was null, the new authorization plugin must be specified in
// config
HiveAuthorizerFactory authorizerFactory = HiveUtils.getAuthorizerFactory(conf ,
HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER);
authorizerV2 = authorizerFactory.createHiveAuthorizer(
new
HiveMetastoreClientFactoryImpl(),
conf, authenticator);
authorizerV2.applyAuthorizationConfigPolicy(conf );
// create the create table grants with new config
createTableGrants = CreateTableAutomaticGrant.create( conf);
}
}
catch
(HiveException e) {
throw
new
RuntimeException(e);
}
if
(LOG.isDebugEnabled()){
Object authorizationClass = getAuthorizationMode() == AuthorizationMode. V1 ?
getAuthorizer() : getAuthorizerV2();
LOG.debug(
"Session is using authorization class "
+ authorizationClass.getClass());
}
return
;
}
|
可以看到在在authorizer 的取值为null时，才会有生产ower权限的操作（CreateTableAutomaticGrant.create(conf)）
authorizer 的值由参数hive.security.authorization.manager获得，在hive0.13里面这个是有默认值的：
|
1
2
|
HIVE_AUTHORIZATION_MANAGER(
"hive.security.authorization.manager"
,
"org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider"
),
|
这样就导致在setupAuth方法中并不会通过CreateTableAutomaticGrant.create(conf)来设置owner的权限，rc找到了。fix也比较简单。
|
1
2
3
4
5
6
7
8
9
10
|
try
{
authenticator = HiveUtils.getAuthenticator( conf,
HiveConf.ConfVars.HIVE_AUTHENTICATOR_MANAGER);
authenticator.setSessionState(
this
);
authorizer = HiveUtils.getAuthorizeProviderManager( conf,
HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER, authenticator,
true
);
createTableGrants = CreateTableAutomaticGrant.create(conf);
//不管authorizer 是否为空都设置相关权限
if
(authorizer ==
null
) {
.......
|
bug fix之后，使用错误的参数，堆栈如下，说明生效了：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
|
14
/
07
/
18
12
:
33
:
38
WARN session.SessionState: authenticator is
null
,
return
,auth has been initialized
14
/
07
/
18
12
:
33
:
38
WARN session.CreateTableAutomaticGrant: Privilege is
null
FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Privilege alldddd is not found.
14
/
07
/
18
12
:
33
:
38
ERROR ql.Driver: FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Privilege alldddd is not found.
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Privilege alldddd is not found.
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:
410
)
at org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:
979
)
at org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:
990
)
at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:
508
)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:
461
)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:
322
)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:
975
)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:
1040
)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:
911
)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:
901
)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:
268
)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:
220
)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:
423
)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:
359
)
at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:
456
)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:
466
)
at org.apache.hadoop.hive.cli.CliDriver.processInitFiles(CliDriver.java:
502
)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:
739
)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:
686
)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:
625
)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39
)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25
)
at java.lang.reflect.Method.invoke(Method.java:
597
)
at org.apache.hadoop.util.RunJar.main(RunJar.java:
208
)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Privilege alldddd is not found.
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.validatePrivilege(CreateTableAutomaticGrant.java:
129
)
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.getGrantorInfoList(CreateTableAutomaticGrant.java:
105
)
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.create(CreateTableAutomaticGrant.java:
57
)
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:
392
)
...
24
more
|
本文转自菜菜光 51CTO博客，原文链接：http://blog.51cto.com/caiguangguang/1440128，如需转载请自行联系原作者
