高并发下hystrix熔断超时及concurrent.RejectedExecutionException: Rejected command because thread-pool queueSiz...

简介: 高并发下hystrix熔断超时及concurrent.RejectedExecutionException: Rejected command because thread-pool queueSiz...

我的一个项目并发200+,观察服务器的日志发现出现了大量的熔断超时偶尔会闪现出RejectedExecutionException: Rejected command because thread-pool queueSize is at rejection threshold。

在高并发的前提下出现熔断超时:

1.先确定是否是自己接口的问题,接口平均响应时长是多少?

举个例子,倘若平均响应时长为200ms,单线程处理的话5次/秒,tomcat最大并发线程数按照100个来算的话,那就是100*5次/秒=500次/秒

那也就是正常情况下,可以承受500次/秒的并发请求。

但是现在200次就不停的刷超时熔断!

2.在接口上使用@HystrixCommand注解少配置了参数

之前是按照下面的方式配置的

@HystrixCommand(fallbackMethod = "your_interface", commandProperties = {
            @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "5000") })

注意这里有个坑:

  • hystrix.command.default.fallback.isolation.semaphore.maxConcurrentRequests 如果并发数达到该设置值,请求会被拒绝和抛出异常并且fallback不会被调用。默认10
  • hystrix.threadpool.default.coreSize 并发执行的最大线程数,默认10
  • hystrix.threadpool.default.maxQueueSize BlockingQueue的最大队列数,
  • hystrix.threadpool.default.queueSizeRejectionThreshold 即使maxQueueSize没有达到,达到queueSizeRejectionThreshold该值后,请求也会被拒绝。

原来是因为这里如果不显示设置coreSize的化,那么就会按照默认的10来进行处理。这就很好理解为什么大量熔断超时了,10-thread*5次/秒/单线程=50次/秒<200次并发量。

 

然后调整参数,最后解决了这个问题:

@HystrixCommand(fallbackMethod = "otaCheckUpdateFallback", commandProperties = {
        @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "5000"),
        }, threadPoolProperties = { @HystrixProperty(name = "coreSize", value = "100"),
        @HystrixProperty(name = "maxQueueSize", value = "20"),
      @HystrixProperty(name = "queueSizeRejectionThreshold", value = "20")})    

同样将参数queueSizeRejectionThreshold调大之后,即便请求频率超过预期,那么当达到queueSizeRejectionThreshold之后会抛出下面的异常。

2018-04-11 20:33:16.294 ERROR [http-nio-9098-exec-30][DirectJDKLog.java:181] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.netflix.hystrix.exception.HystrixRuntimeException: otaCheckUpdate fallback execution rejected.] with root cause
com.netflix.hystrix.exception.HystrixRuntimeException: your_interface fallback execution rejected.
        at com.netflix.hystrix.AbstractCommand.handleFallbackRejectionByEmittingError(AbstractCommand.java:1043) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at com.netflix.hystrix.AbstractCommand.getFallbackOrThrowException(AbstractCommand.java:875) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at com.netflix.hystrix.AbstractCommand.handleThreadPoolRejectionViaFallback(AbstractCommand.java:993) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at com.netflix.hystrix.AbstractCommand.access$400(AbstractCommand.java:60) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:607) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:601) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87) ~[rxjava-1.1.10.jar!/:1.1.10]
        at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$3.onError(AbstractCommand.java:1194) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:54) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
目录
相关文章
|
5天前
服务熔断器-Hystrix
服务熔断器-Hystrix
25 2
|
7月前
|
Java 机器人 Maven
【Java用法】微服务之间的相互调用方式之一,通过FeignClient客户端调用其他微服务的方法包含熔断器(Hystrix)
【Java用法】微服务之间的相互调用方式之一,通过FeignClient客户端调用其他微服务的方法包含熔断器(Hystrix)
89 0
|
5天前
|
Java Sentinel
【熔断限流组件resilience4j和hystrix】
【熔断限流组件resilience4j和hystrix】
|
5天前
|
监控 微服务
Hystrix熔断器设计思想(学习笔记)附(服务监控hystrixDashboard识图)
Hystrix熔断器设计思想(学习笔记)附(服务监控hystrixDashboard识图)
24 0
|
5天前
|
监控 Java Sentinel
springcloud4-服务熔断hystrix及sentinel
springcloud4-服务熔断hystrix及sentinel
27 0
|
9月前
Ribbon、Feign、Hystrix超时&重试&熔断问题
在使用Ribbon、Feign、Hystrix组合时,因为配置的问题出现以下现象,让我的大脑CPU烧的不行不行(拿我老家话说就是“脑子ran滴奥”)
107 0
|
5天前
|
监控 负载均衡 数据可视化
SpringCloud - Hystrix断路器-服务熔断与降级和HystrixDashboard
SpringCloud - Hystrix断路器-服务熔断与降级和HystrixDashboard
31 0
|
5天前
|
监控 数据可视化 Java
【使用Hystrix实现服务容错和熔断】—— 每天一点小知识
【使用Hystrix实现服务容错和熔断】—— 每天一点小知识
|
8月前
|
负载均衡 Java 调度
Hystrix和Sentinel熔断降级设计理念
Hystrix和Sentinel熔断降级设计理念
68 0
|
8月前
|
安全 Java
Hystrix超时机制为服务接口调用超时提供安全保护
Hystrix超时机制为服务接口调用超时提供安全保护
88 1