1. Can't get Kerberos realm
原因分析:
原始代码为:
org.apache.hadoop.security.UserGroupInformation.setConfiguration(conf) sun.security.krb5.Config.refresh()
首先根据传进来的Hadoop配置conf,去设置UserGroupInformation(UGI),方法的调用关系如下(删除了部分不相关代码):
public static void setConfiguration(Configuration conf) { initialize(conf, true); }
initialize方法如下
private static synchronized void initialize(Configuration conf, boolean overrideNameRules) { authenticationMethod = SecurityUtil.getAuthenticationMethod(conf); if (overrideNameRules || !HadoopKerberosName.hasRulesBeenSet()) { try { HadoopKerberosName.setConfiguration(conf); } catch (IOException ioe) { throw new RuntimeException( "Problem with Kerberos auth_to_local name configuration", ioe); } } ...... }
setConfiguration方法如下
public static void setConfiguration(Configuration conf) throws IOException { final String defaultRule; switch (SecurityUtil.getAuthenticationMethod(conf)) { case KERBEROS: case KERBEROS_SSL: try { KerberosUtil.getDefaultRealm(); } catch (Exception ke) { throw new IllegalArgumentException("Can't get Kerberos realm", ke); } ...... } ...... }
getDefaultRealm使用了反射,目的是为了兼容两套jdk,即IBM(com.ibm.security.krb5.internal.Config) 和 Oracle(sun.security.krb5.Config)
public static String getDefaultRealm() throws ClassNotFoundException, NoSuchMethodException, IllegalArgumentException, IllegalAccessException, InvocationTargetException { Object kerbConf; Class<?> classRef; Method getInstanceMethod; Method getDefaultRealmMethod; if (System.getProperty("java.vendor").contains("IBM")) { classRef = Class.forName("com.ibm.security.krb5.internal.Config"); // 获取IBM jdk的类引用 } else { classRef = Class.forName("sun.security.krb5.Config"); // 获取Oracle jdk的类引用 } getInstanceMethod = classRef.getMethod("getInstance", new Class[0]); kerbConf = getInstanceMethod.invoke(classRef, new Object[0]); getDefaultRealmMethod = classRef.getDeclaredMethod("getDefaultRealm", new Class[0]); return (String)getDefaultRealmMethod.invoke(kerbConf, new Object[0]); }
从上述代码来看,先获取Config类引用,然后getInstanceMethod是获得getInstance方法,再次getDefaultRealmMethod是获得getDefaultRealm方法。
因此,假设我们是使用的Oracle的JDK,那么最后是调用的sun.security.krb5.getDefaultRealm()。接下来看一下sun.security.krb5.getDefaultRealm()是如何实现的。
public String getDefaultRealm() throws KrbException { if(this.defaultRealm != null) { // 如果defaultRealm不为空,直接返回defaultRealm return this.defaultRealm; } else { // 如果defaultRealm为null,获取defaultRealm KrbException var1 = null; String var2 = this.getDefault("default_realm", "libdefaults"); if(var2 == null && this.useDNS_Realm()) { try { var2 = this.getRealmFromDNS(); } catch (KrbException var4) { var1 = var4; } } ...... } }
我们假设defaultRealm = null,看一下如何从var2 = this.getRealmFromDNS();来获取defaultRealm
private String getRealmFromDNS() throws KrbException { String var1 = null; String var2 = null; try { var2 = InetAddress.getLocalHost().getCanonicalHostName(); // 1. 获取local host name } catch (UnknownHostException var7) { KrbException var4 = new KrbException(60, "Unable to locate Kerberos realm: " + var7.getMessage()); var4.initCause(var7); throw var4; } String var3 = PrincipalName.mapHostToRealm(var2); // 2. 根据local host name获取realm .... }
mapHostToRealm()方法如下:
static String mapHostToRealm(String var0) { String var1 = null; try { String var2 = null; Config var3 = Config.getInstance(); // 获取Config的单例对象 if((var1 = var3.getDefault(var0, "domain_realm")) != null) { return var1; } ....... } catch (KrbException var5) { ; } return var1; }
这里会获取Config的单例对象,
public static synchronized Config getInstance() throws KrbException { if(singleton == null) { singleton = new Config(); } return singleton; }
再看Config.getInstance();的具体动作就是判断单例对象是否为null,不为null直接返回,为null重新new一个Config对象。
同时,Config类中还有一个方法refresh,其代码如下:
public static synchronized void refresh() throws KrbException { singleton = new Config(); KdcComm.initStatic(); }
从refresh的代码看,只要调用refresh()方法,就会重新生成Config的单例对象。这个refresh()方法,也是我们代码里面要调用的。
再回顾一下我们的原始代码:
org.apache.hadoop.security.UserGroupInformation.setConfiguration(conf) sun.security.krb5.Config.refresh()
回到getInstance()方法,假设singleton单例是null,会生成Config的单例对象。以后,再次调用getInstance方法都会直接返回这个单例对象了,没有再new的机会了。有人开始质疑没有机会new Config()对象了? 调用Config.refresh()方法不是可以new吗? 答案是可以new,但是如果我们的UserGroupInformation.setConfiguration(conf)会抛出异常,是不是Config.refresh()方法就不会被调用了! 我们的错误就是出现在这里,后面分析UserGroupInformation.setConfiguration(conf)怎么抛出异常了。
在我们来看一下new Config()具体做了什么事情。
private Config() throws KrbException { String var1 = getProperty("java.security.krb5.kdc"); // 从系统变量获取kdc地址,假设我们启动JVM时没有设置该变量 if(var1 != null) { this.defaultKDC = var1.replace(':', ' '); } else { this.defaultKDC = null; } this.defaultRealm = getProperty("java.security.krb5.realm"); // 从系统变量获取realm,假设我们启动JVM时也没有设置该变量 if((this.defaultKDC != null || this.defaultRealm == null) && (this.defaultRealm != null || this.defaultKDC == null)) { try { String var3 = this.getJavaFileName(); // 该方法会从JVM参数java.security.krb5.conf以及<java-home>/lib/security/krb5.conf获取到krb5.conf文件 Vector var2; if(var3 != null) { var2 = this.loadConfigFile(var3); this.stanzaTable = this.parseStanzaTable(var2); if(DEBUG) { System.out.println("Loaded from Java config"); } } else { // 假设JVM参数java.security.krb5.conf以及<java-home>/lib/security/krb5.conf都没有获取到krb5.conf文件 boolean var4 = false; if(isMacosLionOrBetter()) { try { this.stanzaTable = SCDynamicStoreConfig.getConfig(); if(DEBUG) { System.out.println("Loaded from SCDynamicStoreConfig"); } var4 = true; } catch (IOException var6) { ; } } if(!var4) { var3 = this.getNativeFileName(); // 我们是centos机器, 会拿到/etc/krb5.conf var2 = this.loadConfigFile(var3); // 加载/etc/krb5.conf文件 this.stanzaTable = this.parseStanzaTable(var2); if(DEBUG) { System.out.println("Loaded from native config"); } } } } catch (IOException var7) { ; } } else { throw new KrbException("System property java.security.krb5.kdc and java.security.krb5.realm both must be set or neither must be set."); } }
我们的问题就出在var2 = this.loadConfigFile(var3); 位置,因为加载/etc/krb5.conf文件的时候,恰好/etc/krb5.conf文件不存在,因为我们会把修改的krb5.conf去替换/etc/krb5.conf文件,在替换的时间内,恰好去loadConfigFile(),该方法就报了FileNotFoundException的异常。这个异常一直throw到UserGroupInformation.setConfiguration(conf)调用的地方,导致我们永远调用不到Config.refresh()方法。
2. 报错com.google.common.util.concurrent.UncheckedTimeoutException: java.util.concurrent.TimeoutException
原因分析:首先这个异常是因为调试上述报错产生的,所以顺便分析下原因。
上述报错是Can't get Kerberos realm,网上查一下,大概是因为拿不到kdc和realm。
因此,我在JVM启动参数中添加了如下3个参数:
-Djava.security.krb5.conf=/etc/krb5.conf \ -Djava.security.krb5.kdc=node1:8080 \ -Djava.security.krb5.realm=KFC.com \
指定了krb5.conf文件,kdc地址,realm值。然后重启程序,发现可以正常使用,然后把/etc/krb5.conf文件删除了(上个错误其实猜想到了是因为读不到krb5.conf造成的)。
程序竟然报错 java.util.concurrent.TimeoutException,打jstack
TimeoutException 的jstack如下: "builtin-checker-serviceId-58" prio=10 tid=0x00007f678800e800 nid=0x4084 waiting for monitor entry [0x00007f672fffe000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1074) - waiting to lock <0x00000000a8b940d0> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation) ...... at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 调用UserGroupInformation.loginUserFromKeytabAndReturnUGI被block了 往上找jstack, "builtin-checker-serviceId-59" prio=10 tid=0x00007f67680b3800 nid=0x4097 runnable [0x00007f672f2ee000] java.lang.Thread.State: RUNNABLE at java.net.PlainDatagramSocketImpl.receive0(Native Method) - locked <0x000000009a0076e0> (a java.net.PlainDatagramSocketImpl) at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:146) - locked <0x000000009a0076e0> (a java.net.PlainDatagramSocketImpl) at java.net.DatagramSocket.receive(DatagramSocket.java:816) - locked <0x000000009a017848> (a java.net.DatagramPacket) - locked <0x000000009a0076a0> (a java.net.DatagramSocket) at sun.security.krb5.internal.UDPClient.receive(NetClient.java:207) // 卡主了 at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:390) at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:343) at java.security.AccessController.doPrivileged(Native Method) at sun.security.krb5.KdcComm.send(KdcComm.java:327) at sun.security.krb5.KdcComm.send(KdcComm.java:219) at sun.security.krb5.KdcComm.send(KdcComm.java:191) at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:319) at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:364) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:735) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687) at javax.security.auth.login.LoginContext.login(LoginContext.java:595) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1092) - locked <0x00000000a8b940d0> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation) ........ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
从jstack中看到UDPClient.receive卡主了,为什么卡主了,不知道! 问大神,大神说加入JVM调试参数-Dsun.security.krb5.debug=true,可以打印日志到console中。在console中看到如下日志:
Ordering keys wrt default_tkt_enctypes list default etypes for default_tkt_enctypes: 3 1 16. default etypes for default_tkt_enctypes: 3 1 16. >>> KrbAsReq creating message >>> KrbKdcReq send: kdc=node1 UDP:88, timeout=30000, number of retries =3, #bytes=134 >>> KDCCommunication: kdc=node1 UDP:88, timeout=30000,Attempt =1, #bytes=134 SocketTimeOutException with attempt: 1 >>> KDCCommunication: kdc=node1 UDP:88, timeout=30000,Attempt =2, #bytes=134 SocketTimeOutException with attempt: 2 >>> KDCCommunication: kdc=node1 UDP:88, timeout=30000,Attempt =3, #bytes=134 SocketTimeOutException with attempt: 3 >>> KrbKdcReq send: error trying node1 java.net.SocketTimeoutException: Receive timed out at java.net.PlainDatagramSocketImpl.receive0(Native Method) at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:146) at java.net.DatagramSocket.receive(DatagramSocket.java:816) at sun.security.krb5.internal.UDPClient.receive(NetClient.java:207) at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:390) at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:343) at java.security.AccessController.doPrivileged(Native Method) at sun.security.krb5.KdcComm.send(KdcComm.java:327) at sun.security.krb5.KdcComm.send(KdcComm.java:219) at sun.security.krb5.KdcComm.send(KdcComm.java:191) at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:319) at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:364) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:735) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687) at javax.security.auth.login.LoginContext.login(LoginContext.java:595) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1092) ........
看到默认去连了KDC的88端口,默认端口被改成了1088,所以连接失败,导致超时。 听说没有参数可以设置KDC的端口, 不知道真假,在-Djava.security.krb5.kdc参数中指定kdc端口无效。
参考: https://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/KerberosReq.html 及源代码