zabbix 突然间报:Zabbix agent on {HOST.NAME} is unreachable for 5 minutes
N多机器都有这样的报警;
登陆查看被监控机,一切正常,网络一切正常;
查看zabbix agent 日志,没有异常;
查看zabbix server 日志 ,大部分提示信息为:
21567:20131203:141448.893 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update ids set nextid=nextid+1 where nodeid=0 and table_name='events' and field_name='eventid']
zabbix_server [21567]: ERROR [file:db.c,line:1501] Something impossible has just happened.
update triggers set lastchange=1386049597,value=1 where triggerid=14912;
update ids set nextid=nextid+1 where nodeid=0 and table_name='events' and field_name='eventid'
delete from escalations where escalationid between 364655 and 364665
登陆MySQL查看,错误日志中的SQL都处在 lock wait 状态,磁盘IO利用率100%,
应该是IO导致的问题,可以从这条语句判断:update triggers set lastchange=1386049597,value=1 where triggerid=14912; 触发器的状态没有更新过来;
IO高的原因是 其他机器在向它scp数据,
数据scp完毕后,zabbix也恢复正常,原以为是zabbixbug,剩下的事情就是优化数据库,修改zabbix的语句!
本文转自 位鹏飞 51CTO博客,原文链接:http://blog.51cto.com/weipengfei/1335388,如需转载请自行联系原作者