Java访问Elasticsearch报错Request cannot be executed; I/O reactor status: STOPPED

本文涉及的产品
检索分析服务 Elasticsearch 版,2核4GB开发者规格 1个月
简介: Java访问Elasticsearch报错Request cannot be executed; I/O reactor status: STOPPED

Java访问Elasticsearch报错Request cannot be executed; I/O reactor status: STOPPED

javayouhuiquan.jpg

简介

使用ES过程中遇到一个Request cannot be executed; I/O reactor status: STOPPED 的异常,大概意思是和server端的连接异常终止了。开始以为是引用的版本不对,或者自己使用问题,后来发现就是因为OOM导致程序宕机,进而引发连接终止。

环境

功能

SpringBoot 的程序通过 SpringDataElasticsearch 访问ES-server 获取数据。

ES-SERVER

  • 版本:7.15.2

ES-CLIENT

ES-CLIENT 就是 SpringBoot 程序,核心pom依赖:

 

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
    <version>2.5.3</version>
</dependency>

问题

  • 部署到测试环境之后,用户量大了之后接口报错,报错信息为:

 

Request cannot be executed; I/O reactor status: STOPPED
  • 后端程序处于假死状态,访问其他接口一直转圈,无法响应内容
  • 重启程序之后一切正常,用户量大了之后又有此问题
  • 详细报错日志为:

 

2021-12-31/19:26:31.748||||^_^|[pool-4-thread-1] ERROR o.a.http.impl.nio.client.InternalHttpAsyncClient 66 - I/O reactor terminated abnormally
org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:359)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.OutOfMemoryError: Java heap space
2021-12-31/19:26:32.783||||^_^|[http-nio-8092-exec-3] ERROR com.nghsmart.datacenter.handler.ExceptionHandler 67 - handleRuntimeException:
java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:888)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654)
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1624)
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1594)
        at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1110)
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.lambda$search$11(ElasticsearchRestTemplate.java:296)
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:383)
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.search(ElasticsearchRestTemplate.java:296)
        at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.search(AbstractElasticsearchTemplate.java:513)
        at com.nghsmart.datacenter.domain.area.service.impl.AreaDashboradServiceImpl.getRotationStatisticsInfo(AreaDashboradServiceImpl.java:166)
        at com.nghsmart.datacenter.domain.area.controller.AreaDashboradController.getRotationStatisticsInfo(AreaDashboradController.java:76)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1064)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at com.nghsmart.commonauthentication.filter.JwtAuthenticationTokenFilter.doFilterInternal(JwtAuthenticationTokenFilter.java:52)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
        at org.apache.http.util.Asserts.check(Asserts.java:46)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:279)
... 69 common frames omitted
  • 测试环境用户量一大就报,本地IDEA中一次也没有报这个问题
  • 使用JMETER设置50个线程并发访问可以稳定复现此问题

原因

程序接口中将一块很大的数据存进JAVA集合中引发了oom,oom异常导致程序宕机,处于假死状态,进而导致 ES-CLIENT 和 ES-SERVER 端的 http 连接异常终止,然后org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning 方法报异常。

ensureRunning:

 

protected void ensureRunning() {
    final Status currentStatus = this.status.get();
    Asserts.check(currentStatus == Status.ACTIVE, "Request cannot be executed; " +
            "I/O reactor status: %s", currentStatus);
}

check:

 

public static void check(final boolean expression, final String message, final Object arg) {
    if (!expression) {
        throw new IllegalStateException(String.format(message, arg));
    }
}
  • SpringDataElasticsearch 和ES-SERVER 是长链接,只要报了OOM,当前和 ES-SERVER 的连接线程都将报异常,也就是说,虽然OOM只报了一次,但是可能有多个线程都在 Asserts.check 方法中报异常
  • 其实不仅仅是和ES-SERVER 的连接异常关闭,观察大量日志后发现和nacos 的连接也有问题:

 

2022-01-04 15:59:07.366 [com.alibaba.nacos.client.remote.worker]  INFO com.alibaba.nacos.common.remote.client.printIfInfoEnabled:60 - [117be10e-119f-4253-a348-71c95a8978a1]Server healthy check fail,currentConnection=1641282430796_192.168.1.90_43108
2022-01-04 15:59:07.367 [com.alibaba.nacos.client.remote.worker]  INFO com.alibaba.nacos.common.remote.client.printIfInfoEnabled:60 - [117be10e-119f-4253-a348-71c95a8978a1] try to re connect to a new server ,server is  not appointed,will choose a random server.
  • 测试环境运行SpringBoot内存配置为:-Xms256m -Xmx512m,使用 java -jar 命令启动。测试环境能复现,本地idea无法复现的原因是内存配置不同导致的,本地idea内存给的2g。编辑 run/debug configuration ,设置 VM options 和测试环境相同,使用jemter压测本地idea中跑的程序,也能复现此问题

解决办法

  • 优化接口,解决OOM问题
  • 增大测试环境内存配置

解决过程

HttpClient问题

这个思路并没有解决我的问题,但是解决问题的过程中花了大量时间研究。其实应该仔细阅读报错log的上下文和接口代码上下文,一开始没有留意到oom的异常

大概意思是说低版本 httpClient 有这个问题,高版本就没有这个问题,是apache-httpclient 导致的。具体是因为捕获到线程异常之后http连接就主动终止了,解决思路是升级版本或者是重写处理器,遇到异常之后不终止连接。

升级httpclient相关pom依赖:

 

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
    <!-- 排除 httpclient、httpcore、 httpcore-nio这三个依赖,并手动引入最新版本,解决es客户端报Request cannot be executed; I/O reactor status: STOPPED问题 -->
    <exclusions>
<!--                <exclusion>-->
<!--                    <groupId>org.apache.httpcomponents</groupId>-->
<!--                    <artifactId>httpclient</artifactId>-->
<!--                </exclusion>-->
<!--                <exclusion>-->
<!--                    <groupId>org.apache.httpcomponents</groupId>-->
<!--                    <artifactId>httpcore</artifactId>-->
<!--                </exclusion>-->
<!--                <exclusion>-->
<!--                    <groupId>org.apache.httpcomponents</groupId>-->
<!--                    <artifactId>httpcore-nio</artifactId>-->
<!--                </exclusion>-->
        <exclusion>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<!-- start -->
<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-client</artifactId>
    <version>7.15.2</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore-nio</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<!--  end  -->
<!-- start:排除 httpclient、httpcore、 httpcore-nio这三个依赖,并手动引入最新版本,解决es客户端报Request cannot be executed; I/O reactor status: STOPPED问题 -->
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.13</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcore-nio</artifactId>
    <version>4.4.14</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcore</artifactId>
    <version>4.4.14</version>
</dependency>
<!--  end   -->

重写ElasticsearchRestTemplate:

 

import com.alibaba.fastjson.JSON;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.Header;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.DefaultConnectionKeepAliveStrategy;
import org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager;
import org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor;
import org.apache.http.message.BasicHeader;
import org.apache.http.nio.reactor.IOReactorException;
import org.apache.http.nio.reactor.IOReactorExceptionHandler;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
@Slf4j
@Configuration
public class EsClientConfig {
    @Value("${es.host}")
    private String host;
    @Value("${es.port}")
    private Integer port;
    @Bean
    public ElasticsearchRestTemplate initElasticsearchRestTemplate() {
        RestClientBuilder builder = RestClient.builder(new HttpHost(host, port))
                .setRequestConfigCallback(
                        config -> config.setConnectTimeout(180000)
                                .setConnectionRequestTimeout(180000)
                                .setSocketTimeout(180000))
                .setHttpClientConfigCallback(
                        httpClientBuilder -> {
                            httpClientBuilder.setMaxConnTotal(100);
                            httpClientBuilder.setMaxConnPerRoute(50);
                            List<Header> headers = new ArrayList<>(2);
                            headers.add(new BasicHeader("Connection", "keep-alive"));
                            headers.add(new BasicHeader("Keep-Alive", "720"));
                            httpClientBuilder.setDefaultHeaders(headers);
                            httpClientBuilder.setKeepAliveStrategy(DefaultConnectionKeepAliveStrategy.INSTANCE);
                            try {
                                DefaultConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
                                ioReactor.setExceptionHandler(new IOReactorExceptionHandler() {
                                    @Override
                                    public boolean handle(IOException e) {
                                        log.debug("System may be unstable: IOReactor encountered a checked exception : " + e.getMessage());
                                        log.debug("start setHttpClientConfigCallback handle IOException e printStackTrace");
                                        e.printStackTrace();
                                        log.debug("end setHttpClientConfigCallback handle IOException e printStackTrace");
                                        // Return true to note this exception as handled, it will not be re-thrown
                                        return true;
                                    }
                                    @Override
                                    public boolean handle(RuntimeException e) {
                                        log.debug("System may be unstable: IOReactor encountered a runtime exception : " + e.getMessage() + ",e is {}", e);
                                        log.debug("start setHttpClientConfigCallback handle RuntimeException e printStackTrace ");
                                        e.printStackTrace();
                                        log.debug("end setHttpClientConfigCallback handle RuntimeException e printStackTrace ");
                                        // Return true to note this exception as handled, it will not be re-thrown
                                        return true;
                                    }
                                });
                                httpClientBuilder.setConnectionManager(new PoolingNHttpClientConnectionManager(ioReactor));
                            } catch (IOReactorException e) {
                                throw new RuntimeException(e);
                            }
                            return httpClientBuilder;
                        }
                );
        ElasticsearchRestTemplate elasticsearchRestTemplate = new ElasticsearchRestTemplate(new RestHighLevelClient(builder));
        return elasticsearchRestTemplate;
    }
}

关于此问题相关参考:

 

__EOF__

 

相关实践学习
使用阿里云Elasticsearch体验信息检索加速
通过创建登录阿里云Elasticsearch集群,使用DataWorks将MySQL数据同步至Elasticsearch,体验多条件检索效果,简单展示数据同步和信息检索加速的过程和操作。
ElasticSearch 入门精讲
ElasticSearch是一个开源的、基于Lucene的、分布式、高扩展、高实时的搜索与数据分析引擎。根据DB-Engines的排名显示,Elasticsearch是最受欢迎的企业搜索引擎,其次是Apache Solr(也是基于Lucene)。 ElasticSearch的实现原理主要分为以下几个步骤: 用户将数据提交到Elastic Search 数据库中 通过分词控制器去将对应的语句分词,将其权重和分词结果一并存入数据 当用户搜索数据时候,再根据权重将结果排名、打分 将返回结果呈现给用户 Elasticsearch可以用于搜索各种文档。它提供可扩展的搜索,具有接近实时的搜索,并支持多租户。
相关文章
|
17天前
|
分布式计算 Java MaxCompute
ODPS MR节点跑graph连通分量计算代码报错java heap space如何解决
任务启动命令:jar -resources odps-graph-connect-family-2.0-SNAPSHOT.jar -classpath ./odps-graph-connect-family-2.0-SNAPSHOT.jar ConnectFamily 若是设置参数该如何设置
|
17天前
|
SQL Java 数据库连接
在Java应用中,数据库访问常成为性能瓶颈。连接池技术通过预建立并复用数据库连接,有效减少连接开销,提升访问效率
在Java应用中,数据库访问常成为性能瓶颈。连接池技术通过预建立并复用数据库连接,有效减少连接开销,提升访问效率。本文介绍了连接池的工作原理、优势及实现方法,并提供了HikariCP的示例代码。
31 3
|
20天前
|
SQL Java 数据库连接
打破瓶颈:利用Java连接池技术提升数据库访问效率
在Java应用中,数据库访问常成为性能瓶颈。连接池技术通过预建立并复用数据库连接,避免了频繁的连接建立和断开,显著提升了数据库访问效率。常见的连接池库包括HikariCP、C3P0和DBCP,它们提供了丰富的配置选项和强大的功能,帮助优化应用性能。
38 2
|
21天前
|
安全 小程序 Java
Java“AccessControlException”报错解决
Java中的“AccessControlException”通常发生在尝试访问受安全策略限制的资源时。解决方法包括:1. 检查安全策略文件(java.policy)配置;2. 确保代码具有足够的权限;3. 调整JVM启动参数以放宽安全限制。
|
30天前
|
Java
Java访问外网图片地址时,如何添加代理?
【10月更文挑战第14天】Java访问外网图片地址时,如何添加代理?
22 2
|
1月前
|
小程序 Java
小程序访问java后台失败解决方案
小程序访问java后台失败解决方案
44 2
|
1月前
|
小程序 JavaScript Java
小程序访问java后台
小程序访问java后台
27 1
|
1月前
|
存储 Java
深入理解java对象的访问定位
这篇文章深入探讨了Java对象的访问定位机制,比较了使用句柄和直接指针两种主流的对象访问方式,并指出了它们各自的优势,例如句柄访问在对象移动时的稳定性和直接指针访问的速度优势。
33 0
深入理解java对象的访问定位
|
2月前
|
域名解析 分布式计算 网络协议
java遍历hdfs路径信息,报错EOFException
java遍历hdfs路径信息,报错EOFException
37 3
|
2月前
|
缓存 Java Linux
java操作hbase报错:KeeperErrorCode=NoNode for /hbase-unsecure/master
java操作hbase报错:KeeperErrorCode=NoNode for /hbase-unsecure/master
139 2