c3p0存在严重bug “APPARENT DEADLOCK“的问题

简介: c3p0存在严重bug “APPARENT DEADLOCK“的问题

开发时用了Spring JdbcTemplate + c3p0组合,做性能测试时出现严重的性能问题。40的并发访问,开始时候系统正常,几分钟后,应用就不能访问了。停止40并发的压力,1分钟左右又可以正常访问应用程序了,log日志:

2011-11-01 13:48:39,378 [com.mchange.v2.async.ThreadPoolAsynchronousRunner:435]  
-[WARN] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@4ec5571b    
 -- APPARENT DEADLOCK!!!     
Complete Status: [num_managed_threads: 10, num_active: 10; activeTasks: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7dd9d603
(com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6),
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@68719f81 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#9),
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7b9f03b8 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7),    

把c3p0换为DBCP后,可以正常进行性能测试,访问正常,判定c3p0存在bug。

c3p0从连接池中获取和释放连接时,采用了异步的处理方式。在调用了从c3p0获得的连接的close方法时,不是立即放回池中的,而是放入一 个事件队列中等待c3p0内部的线程池顺序的处理。

c3p0中的2个跟内部的连接池有关参数:maxAdministrativeTaskTimenumHelperThreads

官方描述在:http://www.mchange.com/projects/c3p0/index.html

maxAdministrativeTaskTime中说c3p0中的很多功能不是由 client threads完成的,是通过内部线程池完成的,如果某些内部功能“挂起”了线程池中的线程,那线程池中的线程将被耗尽,其最后结果就是c3p0停止响应了。

需要内部线程池处理的功能:

expandPool -- 在扩大池中连接的时候,如管理的连接从20扩大到40个  
shrinkPool -- 在缩小连接池的时候,这个过程需要真正关闭一些连接  
doCheckinManaged -- 把连接返回给连接池  
checkIdleResources -- 定时检查池中的连接  

“APPARENT DEADLOCK”警告就是由于c3p0内部线程池的所有线程都被挂起而造成的,因此官方文档说把numHelperThreads设置大一些可以有效避免这个警告。但是这种方法只能延缓问题的出现,并不能避免整个应用被挂起的问题。

查看log日志,会发现哪些任务的线程:

[com.mchange.v2.async.ThreadPoolAsynchronousRunner:435]-[WARN] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@4ec5571b  
 -- APPARENT DEADLOCK!!! Complete Status: [num_managed_threads: 10, num_active: 10;   
 activeTasks:   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7dd9d603 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6),   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@68719f81 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#9),   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7b9f03b8 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7),   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4ad6470 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4),   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@139cf776 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0),   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@263a6e09 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2),   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@460e247a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3),   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4a7ce984 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1),   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4ec6ff50 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5),   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@6aa40597 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#8);   
 pendingTasks:   
 com.mchange.v2.resourcepool.BasicResourcePool$6@366b3333,   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@37ee752e,   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@84f3bb2,   
 com.mchange.v2.resourcepool.BasicResourcePool$6@9d82761,   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4e015653,   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@39ca8b27,   
 com.mchange.v2.resourcepool.BasicResourcePool$6@2075cb15,   
 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1567e059,   
 com.mchange.v2.resourcepool.BasicResourcePool$6@56b9fe09,  

可以看到AcquireTask占用了内部线程池的所有线程,没有线程可以来执行BasicResourcePool$6对应的任务,而这个任务的作用就是把池外使用完的连接放回池内的,c3p0被挂起了。

发生 DEADLOCK 的前提条件:

  1. 连接池会进行expandPool操作,因为这样才能产生AcquireTask对象,这个对象会阻塞内部连接池,因此,如果把minPoolSize和maxPoolSize设置为一样大可以避免这个问题
  2. AcquireTask的执行持续很长时间,要大于10秒,在实际环境中是有可能放生的
  3. acquireIncrement要大于numHelperThreads,这样才有可能把所有内部线程都占用
  4. 连接池的访问压力要很大,压力线程数要大于连接池现有的连接数


相关文章
|
4月前
|
Java 数据库连接 mybatis
项目移植到原先mybasis项目里出现BindingException: Invalid bound statement (not found): **selectPage
项目移植到原先mybasis项目里出现BindingException: Invalid bound statement (not found): **selectPage
45 1
|
10月前
|
Linux 开发工具 Android开发
[√]leak tracer的stack address始终无法被addr2line识别
[√]leak tracer的stack address始终无法被addr2line识别
117 0
|
移动开发 JSON Kubernetes
k8s异常诊断之no space left on device.
某用户反馈,特定节点一直拉不起来pod,提示no space left on device.,手动去docker run也是相同的报错 # docker run --name aestools-perf --cap-add CAP_SYS_ADMIN --privileged -ti --rm registry-vpc.cn-beijing.aliyuncs.com/my-nettools/aestools:onlyperf docker: Error response from daemon: error creating overlay mount to /var/li
2333 3
|
SQL 关系型数据库 MySQL
this is incompatible with sql_mode=only_full_group_by、错误解决方案(亲测可用)
this is incompatible with sql_mode=only_full_group_by、错误解决方案(亲测可用)
3903 0
this is incompatible with sql_mode=only_full_group_by、错误解决方案(亲测可用)
|
iOS开发 Windows
iPhone is busy: Preparing debugger support for iPhone的解决办法
iPhone is busy: Preparing debugger support for iPhone的解决办法
189 0
|
Docker 容器
no space left on device解决方法
no space left on device解决方法
|
关系型数据库 MySQL
MySQL Crash Errcode: 28 - No space left on device
一台MySQL服务器突然Crash了,检查进程 ps -ef | grep -i mysql 发现mysqld进程已经没有了, 检查错误日志时发现MySQL确实Crash了。具体如下所示:     注意日志中的时间: 09:49:52 UTC是UTC时间(协调世界时间) , 加上8小时就是东八区的时间17:45:52,日志前段是UTC时间,后面又是系统时间。
5085 0
|
关系型数据库 Java 数据库
APPARENT DEADLOCK!!! - C3P0连接池DeadLock机制分析
1 问题 近期,刚上线不久的生产系统的数据库连接池 C3P0 (版本为0.9.5.2)突然报出 APPARENT DEADLOCK!!! 错误。 1.1 错误日志 错误日志如下。
5193 0