Pre
JVM-11虚拟机性能监控与故障处理工具之【JDK的可视化工具-JConsole】
jstack
jstack主要用来查看某个Java进程内的线程堆栈信息 ，建议多杀几次线程dump ,每次都有某个事件，基本没跑了~
用法也很简单
Thread dump
Found one Java-level deadlock: ============================= "T-ShortConn-4": waiting to lock monitor 0x00007f87e1d95718 (object 0x00000006cc86c090, a java.lang.Class), which is held by "T-ShortConn-0" "T-ShortConn-0": waiting to lock monitor 0x00007f875c00f188 (object 0x00000006c6f34238, a java.util.concurrent.ConcurrentHashMap), which is held by "localhost-startStop-1" "localhost-startStop-1": waiting to lock monitor 0x00007f87e1d95718 (object 0x00000006cc86c090, a java.lang.Class), which is held by "T-ShortConn-0" Java stack information for the threads listed above: =================================================== "T-ShortConn-4": at com.artisan.bus.tool.redis.RedisClusterHelper.getClient(RedisClusterHelper.java:38) - waiting to lock <0x00000006cc86c090> (a java.lang.Class for com.artisan.bus.tool.redis.RedisClusterHelper) at com.artisan.bus.tool.redis.RedisClusterHelper.listOs(RedisClusterHelper.java:328) at com.artisan.bus.tool.redis.RedisClusterHelper.lSize(RedisClusterHelper.java:193) at com.artisan.bus.service.comunication.CommunicationService.send(CommunicationService.java:1922) at com.artisan.bus.service.comunication.CommunicationService$$FastClassBySpringCGLIB$$cacacc60.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.artisan.bus.service.comunication.CommunicationService$$EnhancerBySpringCGLIB$$8902c097.send(<generated>) at net.artisan.manager.DataProcessorManager.send(DataProcessorManager.java:850) at net.artisan.manager.DataProcessorManager.processShortLinkData(DataProcessorManager.java:833) at net.artisan.common.ShortLinkProcessorThread.run(ShortLinkProcessorThread.java:22) at java.lang.Thread.run(Thread.java:748) "T-ShortConn-0": at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187) - waiting to lock <0x00000006c6f34238> (a java.util.concurrent.ConcurrentHashMap) at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:486) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:389) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1002) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) at com.artisan.bus.tool.utils.SpringApplicationContext.getBean(SpringApplicationContext.java:31) at com.artisan.bus.tool.redis.RedisClusterHelper.getClient(RedisClusterHelper.java:40) - locked <0x00000006cc86c090> (a java.lang.Class for com.artisan.bus.tool.redis.RedisClusterHelper) at com.artisan.bus.tool.redis.RedisClusterHelper.listOs(RedisClusterHelper.java:328) at com.artisan.bus.tool.redis.RedisClusterHelper.lSize(RedisClusterHelper.java:193) at com.artisan.bus.service.comunication.CommunicationService.send(CommunicationService.java:1922) at com.artisan.bus.service.comunication.CommunicationService$$FastClassBySpringCGLIB$$cacacc60.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.artisan.bus.service.comunication.CommunicationService$$EnhancerBySpringCGLIB$$8902c097.send(<generated>) at net.artisan.manager.DataProcessorManager.send(DataProcessorManager.java:850) at net.artisan.manager.DataProcessorManager.processShortLinkData(DataProcessorManager.java:833) at net.artisan.common.ShortLinkProcessorThread.run(ShortLinkProcessorThread.java:22) at java.lang.Thread.run(Thread.java:748) "localhost-startStop-1": at com.artisan.bus.tool.redis.RedisClusterHelper.getClient(RedisClusterHelper.java:38) - waiting to lock <0x00000006cc86c090> (a java.lang.Class for com.artisan.bus.tool.redis.RedisClusterHelper) at com.artisan.bus.tool.redis.RedisClusterHelper.del(RedisClusterHelper.java:63) at com.artisan.bus.dao.impl.settings.template.strategy.StrategyTemplateDao.cacheStrategyTemplate(StrategyTemplateDao.java:173) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy82.cacheStrategyTemplate(Unknown Source) at com.artisan.bus.service.impl.task.CheckTaskService.cacheStrategyTemplate(CheckTaskService.java:3658) at com.artisan.bus.service.impl.task.CheckTaskService$$FastClassBySpringCGLIB$$b794226f.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.artisan.bus.service.impl.task.CheckTaskService$$EnhancerBySpringCGLIB$$7b018424.cacheStrategyTemplate(<generated>) at com.artisan.bus.web.filter.ApplicationConfigue.setApplicationContext(ApplicationConfigue.java:74) at org.springframework.context.support.ApplicationContextAwareProcessor.invokeAwareInterfaces(ApplicationContextAwareProcessor.java:121) at org.springframework.context.support.ApplicationContextAwareProcessor.postProcessBeforeInitialization(ApplicationContextAwareProcessor.java:97) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) - locked <0x00000006c6f34238> (a java.util.concurrent.ConcurrentHashMap) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542) - locked <0x00000006c6f655a8> (a java.lang.Object) at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:668) at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:634) at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:682) at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:553) at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:494) at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:138) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1269) - locked <0x00000006c6d54a78> (a org.apache.catalina.core.StandardWrapper) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) - locked <0x00000006c6d54a78> (a org.apache.catalina.core.StandardWrapper) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) - locked <0x00000006c6d54a78> (a org.apache.catalina.core.StandardWrapper) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) - locked <0x00000006c54f5f78> (a org.apache.catalina.core.StandardContext) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) - locked <0x00000006c54f5f78> (a org.apache.catalina.core.StandardContext) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260) at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Found 1 deadlock.
Dead Lock 分析
jstack 杀出来的线程堆栈， 划重点~~~
Found one Java-level deadlock: ============================= "T-ShortConn-4": waiting to lock monitor 0x00007f87e1d95718 (object 0x00000006cc86c090, a java.lang.Class), which is held by "T-ShortConn-0" "T-ShortConn-0": waiting to lock monitor 0x00007f875c00f188 (object 0x00000006c6f34238, a java.util.concurrent.ConcurrentHashMap), which is held by "localhost-startStop-1" "localhost-startStop-1": waiting to lock monitor 0x00007f87e1d95718 (object 0x00000006cc86c090, a java.lang.Class), which is held by "T-ShortConn-0" Java stack information for the threads listed above: ===================================================
三个线程 T-ShortConn-4 、 T-ShortConn-0 和 localhost-startStop-1
看括号里面的 object xxxxxxx
T-ShortConn-4 想去获取lock 【object 0x00000006cc86c090】 , 但这个lock被 T-ShortConn-0持有
T-ShortConn-0 想去获取lock 【object 0x00000006c6f34238】 , 但这个lock被 localhost-startStop-1持有
localhost-startStop-1 想去获取lock 【object 0x00000006cc86c090】 , 但这个lock被 T-ShortConn-0持有
====》得出结论
T-ShortConn-0 持有的lock 【0x00000006cc86c090】 被 T-ShortConn-4 和 localhost-startStop-1 需要
localhost-startStop-1 持有的lock 【0x00000006c6f34238】 被 T-ShortConn-0 需要
这不就产生deal lock了么？
分析代码
T-ShortConn-0 持有的lock 【0x00000006cc86c090】 那就去dump中查找这个关键字吧
at com.artisan.bus.tool.utils.SpringApplicationContext.getBean(SpringApplicationContext.java:31) at com.artisan.bus.tool.redis.RedisClusterHelper.getClient(RedisClusterHelper.java:40) - locked <0x00000006cc86c090> (a java.lang.Class for com.artisan.bus.tool.redis.RedisClusterHelper)
localhost-startStop-1 持有的lock 【0x00000006c6f34238】
DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187) 返回的是一个 ConcurrentHashMap ， lock类型就是 ConcurrentHashMap。
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) - locked <0x00000006c6f34238> (a java.util.concurrent.ConcurrentHashMap)
解决
解决方法： 提前初始化RedisClusterHelper这个单例
