Linux集群和自动化维2.6.4 开发类脚本-阿里云开发者社区

开发者社区> 华章出版社> 正文

Linux集群和自动化维2.6.4 开发类脚本

简介:

2.6.4 开发类脚本


业务需求在不断地变化,有时候互联网上的开源方案并不能全部解决,这个时候就需要自己写一些开发类的脚本来满足工作中的需求了,虽然很多时候脚本都可以独立运行,但笔者的做法还是尽量将其return结果写成Nagios能够识别的格式,以便配合Nagios发送报警邮件和信息。

1.监测redis是否正常运行

笔者接触的线上NoSQL业务主要是redis数据库,多用于处理大量数据的高访问负载需求。为了最大化地利用资源,每个redis实例分配的内存并不是很大,有时候程序组的同事导入数据量大的IP list时会导致redis实例崩溃,所以笔者开发了一个redis监测脚本并配合Nagios进行工作,脚本内容如下所示(此脚本在Amazon Linux AMI x86_64下已测试通过):

#!/usr/bin/python

#Check redis Nagios Plungin,Please install the redis-py module.

import redis

import sys

 

STATUS_OK = 0

STATUS_WARNING = 1

STATUS_CRITICAL = 2

 

HOST = sys.argv[1]

PORT = int(sys.argv[2])

WARNING = float(sys.argv[3])

CRITICAL = float(sys.argv[4])

 

def connect_redis(host, port):

    r = redis.Redis(host, port, socket_timeout = 5, socket_connect_timeout = 5)

    return r

 

def main():

    r = connect_redis(HOST, PORT)

    try:

        r.ping()

    except:

        print HOST,PORT,'down'

        sys.exit(STATUS_CRITICAL)

 

    redis_info = r.info()

    used_mem = redis_info['used_memory']/1024/1024/1024.0

    used_mem_human = redis_info['used_memory_human']

 

    if WARNING <= used_mem < CRITICAL:

        print HOST,PORT,'use memory warning',used_mem_human

        sys.exit(STATUS_WARNING)

    elif used_mem >= CRITICAL:

        print HOST,PORT,'use memory critical',used_mem_human

        sys.exit(STATUS_CRITICAL)

    else:

        print HOST,PORT,'use memory ok',used_mem_human

        sys.exit(STATUS_OK)

 

if __name__ == '__main__':

    main()

2.监测机器的IP连接数

需求其实比较简单,先统计IP连接数,如果ip_conns值小于15 000则显示为正常,介于15 000至20 000之间为警告,如果超过20 000则报警,脚本内容如下所示(此脚本在Amazon Linux AMI x86_64下已测试通过):

#!/bin/bash

#Nagios plugin For ip connects

#$1 = 15000 $2 = 20000

ip_conns=`netstat -an | grep tcp | grep EST | wc -l`

messages=`netstat -ant | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'|tr -s '\n' ',' | sed -r 's/(.*),/\1\n/g' `

 

if [ $ip_conns -lt $1 ]

then

    echo "$messages,OK -connect counts is $ip_conns"

    exit 0

fi

if [ $ip_conns -gt $1 -a $ip_conns -lt $2 ]

then

    echo "$messages,Warning -connect counts is $ip_conns"

    exit 1

fi

if [ $ip_conns -gt $2 ]

then

    echo "$messages,Critical -connect counts is $ip_conns"

    exit 2

fi

3.监测机器的CPU利用率脚本

线上的bidder业务机器,在业务繁忙的高峰期会出现CPU利用率达到100%(sys%+ user%),导致后面的流量打在上面却完全进不去的情况,但此时机器、系统负载及Nginx+Lua进程都是完全正常的,所以这种情况下需要开发一个CPU利用率脚本,在超过自定义阀值时报警,方便运维人员批量添加bidder AMI机器以应对峰值,AWS EC2实例机器是可以以小时来计费的,大家在这里也要注意分清系统负载和CPU利用率之间的区别。脚本内容如下所示(此脚本在Amazon Linux AMI x86_64下已测试通过):

#!/bin/bash

# ==============================================================================

# CPU Utilization Statistics plugin for Nagios

#

# USAGE     :   ./check_cpu_utili.sh [-w <user,system,iowait>] [-c <user,system,iowait>] ( [ -i <intervals in second> ] [ -n <report number> ])

#          

# Exemple: ./check_cpu_utili.sh

#          ./check_cpu_utili.sh -w 70,40,30 -c 90,60,40

#          ./check_cpu_utili.sh -w 70,40,30 -c 90,60,40 -i 3 -n 5

#-------------------------------------------------------------------------------

# Paths to commands used in this script.  These may have to be modified to match your system setup.

IOSTAT="/usr/bin/iostat"

 

# Nagios return codes

STATE_OK=0

STATE_WARNING=1

STATE_CRITICAL=2

STATE_UNKNOWN=3

 

# Plugin parameters value if not define

LIST_WARNING_THRESHOLD="70,40,30"

LIST_CRITICAL_THRESHOLD="90,60,40"

INTERVAL_SEC=1

NUM_REPORT=1

# Plugin variable description

PROGNAME=$(basename $0)

 

if [ ! -x $IOSTAT ]; then

    echo "UNKNOWN: iostat not found or is not executable by the nagios user."

    exit $STATE_UNKNOWN

fi

 

print_usage() {

        echo ""

        echo "$PROGNAME $RELEASE - CPU Utilization check script for Nagios"

        echo ""

        echo "Usage: check_cpu_utili.sh -w -c (-i -n)"

        echo ""

        echo "  -w  Warning threshold in % for warn_user,warn_system,warn_iowait CPU (default : 70,40,30)"

        echo "  Exit with WARNING status if cpu exceeds warn_n"

        echo "  -c  Critical threshold in % for crit_user,crit_system,crit_iowait CPU (default : 90,60,40)"

        echo "  Exit with CRITICAL status if cpu exceeds crit_n"

        echo "  -i  Interval in seconds for iostat (default : 1)"

        echo "  -n  Number report for iostat (default : 3)"

        echo "  -h  Show this page"

        echo ""

    echo "Usage: $PROGNAME"

    echo "Usage: $PROGNAME --help"

    echo ""

    exit 0

}

 

print_help() {

    print_usage

        echo ""

        echo "This plugin will check cpu utilization (user,system,CPU_Iowait in %)"

        echo ""

    exit 0

}

 

# Parse parameters

while [ $# -gt 0 ]; do

    case "$1" in

        -h | --help)

            print_help

            exit $STATE_OK

            ;;

        -v | --version)

                print_release

                exit $STATE_OK

                ;;

        -w | --warning)

                shift

                LIST_WARNING_THRESHOLD=$1

                ;;

        -c | --critical)

               shift

                LIST_CRITICAL_THRESHOLD=$1

                ;;

        -i | --interval)

               shift

               INTERVAL_SEC=$1

                ;;

        -n | --number)

               shift

               NUM_REPORT=$1

                ;;       

        *)  echo "Unknown argument: $1"

            print_usage

            exit $STATE_UNKNOWN

            ;;

        esac

shift

done

 

# List to Table for warning threshold (compatibility with

TAB_WARNING_THRESHOLD=(`echo $LIST_WARNING_THRESHOLD | sed 's/,/ /g'`)

if [ "${#TAB_WARNING_THRESHOLD[@]}" -ne "3" ]; then

    echo "ERROR : Bad count parameter in Warning Threshold"

    exit $STATE_WARNING

else 

USER_WARNING_THRESHOLD=`echo ${TAB_WARNING_THRESHOLD[0]}`

SYSTEM_WARNING_THRESHOLD=`echo ${TAB_WARNING_THRESHOLD[1]}`

IOWAIT_WARNING_THRESHOLD=`echo ${TAB_WARNING_THRESHOLD[2]}`

fi

 

# List to Table for critical threshold

TAB_CRITICAL_THRESHOLD=(`echo $LIST_CRITICAL_THRESHOLD | sed 's/,/ /g'`)

if [ "${#TAB_CRITICAL_THRESHOLD[@]}" -ne "3" ]; then

    echo "ERROR : Bad count parameter in CRITICAL Threshold"

    exit $STATE_WARNING

else

USER_CRITICAL_THRESHOLD=`echo ${TAB_CRITICAL_THRESHOLD[0]}`

SYSTEM_CRITICAL_THRESHOLD=`echo ${TAB_CRITICAL_THRESHOLD[1]}`

IOWAIT_CRITICAL_THRESHOLD=`echo ${TAB_CRITICAL_THRESHOLD[2]}`

fi

 

if [ ${TAB_WARNING_THRESHOLD[0]} -ge ${TAB_CRITICAL_THRESHOLD[0]} -o ${TAB_WARNING_THRESHOLD[1]} -ge ${TAB_CRITICAL_THRESHOLD[1]} -o ${TAB_WARNING_THRESHOLD[2]} -ge ${TAB_CRITICAL_THRESHOLD[2]} ]; then

  echo "ERROR : Critical CPU Threshold lower as Warning CPU Threshold "

  exit $STATE_WARNING

fi

 

CPU_REPORT=`iostat -c $INTERVAL_SEC $NUM_REPORT | sed -e 's/,/./g' | tr -s ' ' ';' | sed '/^$/d' | tail -1`

CPU_REPORT_SECTIONS=`echo ${CPU_REPORT} | grep ';' -o | wc -l`

CPU_USER=`echo $CPU_REPORT | cut -d ";" -f 2`

CPU_SYSTEM=`echo $CPU_REPORT | cut -d ";" -f 4`

CPU_IOWAIT=`echo $CPU_REPORT | cut -d ";" -f 5`

CPU_STEAL=`echo $CPU_REPORT | cut -d ";" -f 6`

CPU_IDLE=`echo $CPU_REPORT | cut -d ";" -f 7`

NAGIOS_STATUS="user=${CPU_USER}%,system=${CPU_SYSTEM}%,iowait=${CPU_IOWAIT}%,idle=${CPU_IDLE}%"

NAGIOS_DATA="CpuUser=${CPU_USER};${TAB_WARNING_THRESHOLD[0]};${TAB_CRITICAL_THRESHOLD[0]};0"

 

CPU_USER_MAJOR=`echo $CPU_USER| cut -d "." -f 1`

CPU_SYSTEM_MAJOR=`echo $CPU_SYSTEM | cut -d "." -f 1`

CPU_IOWAIT_MAJOR=`echo $CPU_IOWAIT | cut -d "." -f 1`

CPU_IDLE_MAJOR=`echo $CPU_IDLE | cut -d "." -f 1`

 

# Return

if [ ${CPU_USER_MAJOR} -ge $USER_CRITICAL_THRESHOLD ]; then

        echo "CPU STATISTICS OK:${NAGIOS_STATUS} | CPU_USER=${CPU_USER}%;70;90;0;100"

        exit $STATE_CRITICAL

    elif [ ${CPU_SYSTEM_MAJOR} -ge $SYSTEM_CRITICAL_THRESHOLD ]; then

        echo "CPU STATISTICS OK:${NAGIOS_STATUS} | CPU_USER=${CPU_USER}%;70;90;0;100"

        exit $STATE_CRITICAL

    elif [ ${CPU_IOWAIT_MAJOR} -ge $IOWAIT_CRITICAL_THRESHOLD ]; then

        echo "CPU STATISTICS OK:${NAGIOS_STATUS} | CPU_USER=${CPU_USER}%;70;90;0;100"

        exit $STATE_CRITICAL

    elif [ ${CPU_USER_MAJOR} -ge $USER_WARNING_THRESHOLD ] && [ ${CPU_USER_MAJOR} -lt $USER_CRITICAL_THRESHOLD ]; then

        echo "CPU STATISTICS OK:${NAGIOS_STATUS} | CPU_USER=${CPU_USER}%;70;90;0;100"

        exit $STATE_WARNING

      elif [ ${CPU_SYSTEM_MAJOR} -ge $SYSTEM_WARNING_THRESHOLD ] && [ ${CPU_SYSTEM_MAJOR} -lt $SYSTEM_CRITICAL_THRESHOLD ]; then

        echo "CPU STATISTICS OK:${NAGIOS_STATUS} | CPU_USER=${CPU_USER}%;70;90;0;100"

        exit $STATE_WARNING

      elif  [ ${CPU_IOWAIT_MAJOR} -ge $IOWAIT_WARNING_THRESHOLD ] && [ ${CPU_IOWAIT_MAJOR} -lt $IOWAIT_CRITICAL_THRESHOLD ]; then

        echo "CPU STATISTICS OK:${NAGIOS_STATUS} | CPU_USER=${CPU_USER}%;70;90;0;100"

        exit $STATE_WARNING

else

      

        echo "CPU STATISTICS OK:${NAGIOS_STATUS} | CPU_USER=${CPU_USER}%;70;90;0;100"

        exit $STATE_OK

fi

此脚本参考了Nagios的官方文档https://exchange.nagios.org/并进行了代码精简和移值,源代码是运行在ksh下面的,这里将其移植到了bash下面,ksh下定义数组的方式跟bash还是有区别的;另外有一点也请大家注意,Shell本身是不支持浮点运算的,但可以通过bc或awk的方式来处理。

另外,若要配合PNP4nagios出图(PNP4nagios可以观察一段周期内的CPU利用率峰值),此脚本还可以更精简,脚本内容如下所示(此脚本在Amazon Linux AMI x86_64下已测试通过):

#!/bin/bash

# Nagios return codes

STATE_OK=0

STATE_WARNING=1

STATE_CRITICAL=2

STATE_UNKNOWN=3

 

# Plugin parameters value if not define

LIST_WARNING_THRESHOLD="90"

LIST_CRITICAL_THRESHOLD="95"

INTERVAL_SEC=1

NUM_REPORT=5

 

CPU_REPORT=`iostat -c $INTERVAL $NUM_REPORT  | sed -e 's/,/./g' | tr -s ' ' ';' | sed '/^$/d' |tail -1`

CPU_REPORT_SECTIONS=`echo ${CPU_REPORT} | grep ';' -o | wc -l`

CPU_USER=`echo $CPU_REPORT | cut -d ";" -f 2`

CPU_SYSTEM=`echo $CPU_REPORT | cut -d ";" -f 4`

# Add for integer shell issue

CPU_USER_MAJOR=`echo $CPU_USER | cut -d "." -f 1`

CPU_SYSTEM_MAJOR=`echo $CPU_SYSTEM | cut -d "." -f 1`

CPU_UTILI_COU=`echo ${CPU_USER} + ${CPU_SYSTEM}|bc`

CPU_UTILI_COUNTER=`echo $CPU_UTILI_COU | cut -d "." -f 1`

 

# Return

if [ ${CPU_UTILI_COUNTER} -lt ${LIST_WARNING_THRESHOLD} ]

then

    echo "OK - CPUCOU=${CPU_UTILI_COU}% | CPUCOU=${CPU_UTILI_COU}%;80;90"

    exit ${STATE_OK}

fi

if [ ${CPU_UTILI_COUNTER} -gt ${LIST_WARNING_THRESHOLD} -a ${CPU_UTILI_COUNTER} -lt ${LIST_CRITICAL_THRESHOLD} ]

then

    echo "Warning - CPUCOU=${CPU_UTILI_COUNTER}% | CPUCOU=${CPU_UTILI_COUNTER}%;80;90"

    exit ${STATE_WARNING}

fi

if [ ${CPU_UTILI_COUNTER} -gt ${LIST_CRITICAL_THRESHOLD} ]

then

   echo "Critical - CPUCOU=${CPU_UTILI_COUNTER}% | CPUCOU=${CPU_UTILI_COUNTER}%;80;90"

    exit ${STATE_CRITICAL}

fi

版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

分享:

华章出版社

官方博客
官网链接