logstash问题记录:Attempted to resurrect connection to dead ES instance, but got an error

本文涉及的产品
日志服务 SLS,月写入数据量 50GB 1个月
简介: logstash问题记录:Attempted to resurrect connection to dead ES instance, but got an error

问题描述



发现kibana中日志不在更新了,停留在了一个时间点没有继续更新,然后就去检查各个服务,发现es、logstash、kibana均正常,不过发现es里面没有新日志产生,初步怀疑是logstash没有成功推送过来日志。


问题解决



1.检查logstash日志,logstahs是java开发的,笔者将日志记录到了nohup.out文件中

检查日志文件发现错误:Attempted to resurrect connection to dead ES instance, but got an error


2.然后再次检查es服务健康度,发现es的三个结点均无问题。


3.再次怀疑是logstash问题,发现logstash接收日志都正常,只是将日志输送到es时出了问题,最后还是怀疑问题出在了es。


4.然后c了一下,有网友提供修改es索引数量限制的配置,笔者尝试了下,发现不行。


5.检查es日志(elasticsearch.yml里面配置的日志路径),发现日志里报出了大量的overheap,虽然不是OOM,但是猜测还是和jvm的对大小配置有关,尝试将所有结点的堆容量提升了一倍,栈大小提升了9倍。配置如下:

fed430e6accf43f38d7c4b4f4b95c1a9.png

-Xms4g
-Xmx4g
-Xss10m


6.然后重启了es三个结点、logstash、kibana,之后恢复正常。


问题总结



定位问题,排除了所有不可能,即使剩下的选项再不可能,那问题也一定是在他身上。大胆假设小心求证,就能顺利解决问题。

相关实践学习
日志服务之使用Nginx模式采集日志
本文介绍如何通过日志服务控制台创建Nginx模式的Logtail配置快速采集Nginx日志并进行多维度分析。
相关文章
|
Web App开发 前端开发
【前端异常】Unchecked runtime.lastError: Could not establish connection. Receiving end does not exist.
【前端异常】Unchecked runtime.lastError: Could not establish connection. Receiving end does not exist.
751 0
|
5月前
|
存储
Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index r【已解决】
Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index r【已解决】
138 1
|
5月前
|
JSON 数据格式
【ERROR】Error: transaction invalidated with status (ENDORSEMENT_POLICY_FAILURE)
【ERROR】Error: transaction invalidated with status (ENDORSEMENT_POLICY_FAILURE)
45 0
|
网络安全 Docker 容器
failed to create network error response from daemon filed to setup ip tables问题
failed to create network error response from daemon filed to setup ip tables问题
204 0
|
Linux
七个办法只有一个有效:200 PORT command successful. Consider using PASV.425 Failed to establish connection.
七个办法只有一个有效:200 PORT command successful. Consider using PASV.425 Failed to establish connection.
616 0
七个办法只有一个有效:200 PORT command successful. Consider using PASV.425 Failed to establish connection.
|
索引
ES报错:“type“=>“cluster_block_exception“, “reason“=>“blocked by: [FORBIDDEN/12/index read-only / allow
ES报错:“type“=>“cluster_block_exception“, “reason“=>“blocked by: [FORBIDDEN/12/index read-only / allow
359 0
ES报错:“type“=>“cluster_block_exception“, “reason“=>“blocked by: [FORBIDDEN/12/index read-only / allow
|
存储 NoSQL Linux
redis异常 Commands that may modify the data set are disabled, because this instance is
MISCONF Redis配置为保存RDB快照,但目前无法在磁盘上持久化。可能修改数据集的命令被禁用,因为该实例被配置为在RDB快照失败时报告错误(stop-write -on-bgsave-error选项)。请检查Redis的日志RDB错误的详细信息.
redis异常 Commands that may modify the data set are disabled, because this instance is
|
SQL 数据库 Windows
SQL Server 2005 sp_send_dbmail出现Internal error at FormatRowset (Reason: Not enough storage is available to complete this operation)
案例环境:   操作系统: Windows 2003 SE 32bit(SP2) 数据库版本:Microsoft SQL Server 2005 - 9.00.5069.00 (Intel X86)             Aug 22 2012 16:01:52           ...
1422 0
|
网络协议 Java 容器
【报错】项目启动,仅仅报错 One or more listeners failed to start. Full details will be found in the appropriate container log file
今天spring4.3.13 项目,整合ActiveMQ的时候,项目启动在自动部署到tomcat下的时候,不能正常的部署,仅仅报错如下: Connected to server [2017-12-08 03:57:12,428] Artifact springDemo:war exploded: Artifact is being deployed, please wait.
6317 0