Spring Cloud Hystrix ThreadPool的bug

简介: Spring Cloud Hystrix ThreadPool的bug

BUG背景


JDK: 11.0.4 Spring Cloud Finchley.SR3

相关配置:

#开启hystrix
feign.hystrix.enabled=true
#关闭断路器
hystrix.command.default.circuitBreaker.enabled=false
#禁用hystrix远程调用超时时间
hystrix.command.default.execution.timeout.enabled=false
hystrix.threadpool.default.coreSize=50

Hystrix隔离策略:线程隔离

在Feign调用的时候,会报错:

Caused by: java.util.concurrent.ExecutionException: Observable onError
        at rx.internal.operators.BlockingOperatorToFuture$2.getValue(BlockingOperatorToFuture.java:118) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.internal.operators.BlockingOperatorToFuture$2.get(BlockingOperatorToFuture.java:102) ~[rxjava-1.3.8.jar!/:1.3.8]
        at com.netflix.hystrix.HystrixCommand$4.get(HystrixCommand.java:423) ~[hystrix-core-1.5.18.jar!/:1.5.18]
        at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344) ~[hystrix-core-1.5.18.jar!/:1.5.18]
        ... 30 more
Caused by: java.lang.IllegalArgumentException
        at java.util.concurrent.ThreadPoolExecutor.setCorePoolSize(ThreadPoolExecutor.java:1535) ~[?:?]
        at com.netflix.hystrix.HystrixThreadPool$HystrixThreadPoolDefault.touchConfig(HystrixThreadPool.java:230) ~[hystrix-core-1.5.18.jar!/:1.5.18]
        at com.netflix.hystrix.HystrixThreadPool$HystrixThreadPoolDefault.getScheduler(HystrixThreadPool.java:205) ~[hystrix-core-1.5.18.jar!/:1.5.18]
        at com.netflix.hystrix.AbstractCommand.executeCommandWithSpecifiedIsolation(AbstractCommand.java:710) ~[hystrix-core-1.5.18.jar!/:1.5.18]
        at com.netflix.hystrix.AbstractCommand.executeCommandAndObserve(AbstractCommand.java:638) ~[hystrix-core-1.5.18.jar!/:1.5.18]
        at com.netflix.hystrix.AbstractCommand.applyHystrixSemantics(AbstractCommand.java:546) ~[hystrix-core-1.5.18.jar!/:1.5.18]
        at com.netflix.hystrix.AbstractCommand.access$200(AbstractCommand.java:60) ~[hystrix-core-1.5.18.jar!/:1.5.18]
        at com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:419) ~[hystrix-core-1.5.18.jar!/:1.5.18]
        at com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:413) ~[hystrix-core-1.5.18.jar!/:1.5.18]
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[rxjava-1.3.8.jar!/:1.3.8]
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.3.8.jar!/:1.3.8]

重启有时候复现,有时候不复现,估计是和类加载还有初始化顺序有关的问题。


问题定位


查看问题源代码, 对于每个微服务调用,初始化Hytrix线程池,动态读取配置:

private void touchConfig() {
    final int dynamicCoreSize = properties.coreSize().get();
    final int configuredMaximumSize = properties.maximumSize().get();
    int dynamicMaximumSize = properties.actualMaximumSize();
    final boolean allowSizesToDiverge = properties.getAllowMaximumSizeToDivergeFromCoreSize().get();
    boolean maxTooLow = false;
    if (allowSizesToDiverge && configuredMaximumSize < dynamicCoreSize) {
        //if user sets maximum < core (or defaults get us there), we need to maintain invariant of core <= maximum
        dynamicMaximumSize = dynamicCoreSize;
        maxTooLow = true;
    }
    // In JDK 6, setCorePoolSize and setMaximumPoolSize will execute a lock operation. Avoid them if the pool size is not changed.
    if (threadPool.getCorePoolSize() != dynamicCoreSize || (allowSizesToDiverge && threadPool.getMaximumPoolSize() != dynamicMaximumSize)) {
        if (maxTooLow) {
            logger.error("Hystrix ThreadPool configuration for : " + metrics.getThreadPoolKey().name() + " is trying to set coreSize = " +
                    dynamicCoreSize + " and maximumSize = " + configuredMaximumSize + ".  Maximum size will be set to " +
                    dynamicMaximumSize + ", the coreSize value, since it must be equal to or greater than the coreSize value");
        }
        threadPool.setCorePoolSize(dynamicCoreSize);
        threadPool.setMaximumPoolSize(dynamicMaximumSize);
    }
    threadPool.setKeepAliveTime(properties.keepAliveTimeMinutes().get(), TimeUnit.MINUTES);
}

报错是在threadPool.setCorePoolSize(dynamicCoreSize);,看下为何:

public void setCorePoolSize(int corePoolSize) {
        if (corePoolSize < 0 || maximumPoolSize < corePoolSize)
            throw new IllegalArgumentException();
    //忽略其他代码
}

发现还有maximumPoolSize < corePoolSize的判断,所以,需要先设置MaximumPoolSize,之后设置CorePoolSize,这样可以避免这个问题


相关ISSUES与解决方案


看了下社区,已经有ISSUE了:https://github.com/Netflix/Hystrix/issues/1874 还有对应的PULL REQUEST:https://github.com/Netflix/Hystrix/pull/1877

但是目前,还没有合并,只好自己改了,项目中增加同名同路径替换类,修改:

//jdk9以上的版本,设置coreSize会验证coreSize必须小于maxSize,所以先改变max再设置core
                threadPool.setMaximumPoolSize(dynamicMaximumSize);
                threadPool.setCorePoolSize(dynamicCoreSize);
相关文章
|
8月前
|
监控 负载均衡 Java
深入理解Spring Cloud中的服务网关
深入理解Spring Cloud中的服务网关
|
7月前
|
Java UED 开发者
Spring Boot 降级功能的神秘面纱:Hystrix 与 Resilience4j 究竟藏着怎样的秘密?
【8月更文挑战第29天】在分布式系统中,服务稳定性至关重要。为应对故障,Spring Boot 提供了 Hystrix 和 Resilience4j 两种降级工具。Hystrix 作为 Netflix 的容错框架,通过隔离依赖、控制并发及降级机制增强系统稳定性;Resilience4j 则是一个轻量级库,提供丰富的降级策略。两者均可有效提升系统可靠性,具体选择取决于需求与场景。在面对服务故障时,合理运用这些工具能确保系统基本功能正常运作,优化用户体验。以上简介包括了两个工具的简单示例代码,帮助开发者更好地理解和应用。
160 0
|
3月前
|
消息中间件 监控 Java
如何将Spring Boot + RabbitMQ应用程序部署到Pivotal Cloud Foundry (PCF)
如何将Spring Boot + RabbitMQ应用程序部署到Pivotal Cloud Foundry (PCF)
56 6
|
3月前
|
Java 关系型数据库 MySQL
如何将Spring Boot + MySQL应用程序部署到Pivotal Cloud Foundry (PCF)
如何将Spring Boot + MySQL应用程序部署到Pivotal Cloud Foundry (PCF)
92 5
|
3月前
|
缓存 监控 Java
如何将Spring Boot应用程序部署到Pivotal Cloud Foundry (PCF)
如何将Spring Boot应用程序部署到Pivotal Cloud Foundry (PCF)
73 5
|
8月前
|
资源调度 Java 调度
Spring Cloud Alibaba 集成分布式定时任务调度功能
Spring Cloud Alibaba 发布了 Scheduling 任务调度模块 [#3732]提供了一套开源、轻量级、高可用的定时任务解决方案,帮助您快速开发微服务体系下的分布式定时任务。
15312 34
|
6月前
|
XML 监控 Java
Spring Cloud全解析:熔断之Hystrix简介
Hystrix 是由 Netflix 开源的延迟和容错库,用于提高分布式系统的弹性。它通过断路器模式、资源隔离、服务降级及限流等机制防止服务雪崩。Hystrix 基于命令模式,通过 `HystrixCommand` 封装对外部依赖的调用逻辑。断路器能在依赖服务故障时快速返回备选响应,避免长时间等待。此外,Hystrix 还提供了监控功能,能够实时监控运行指标和配置变化。依赖管理方面,可通过 `@EnableHystrix` 启用 Hystrix 支持,并配置全局或局部的降级策略。结合 Feign 可实现客户端的服务降级。
336 23
|
6月前
|
Java 对象存储 开发者
故障隔离与容错处理:Hystrix在Spring Cloud和Netflix OSS中的应用
故障隔离与容错处理:Hystrix在Spring Cloud和Netflix OSS中的应用
90 3
|
8月前
|
负载均衡 Java Spring
Spring cloud gateway 如何在路由时进行负载均衡
Spring cloud gateway 如何在路由时进行负载均衡
842 15
|
8月前
|
Java Spring
spring cloud gateway在使用 zookeeper 注册中心时,配置https 进行服务转发
spring cloud gateway在使用 zookeeper 注册中心时,配置https 进行服务转发
190 3