巧用shell+rsync实现日志自动过滤处理压缩上传分类(在斯凯的最终版本下周离职)-阿里云开发者社区

巧用shell+rsync实现日志自动过滤处理压缩上传分类(在斯凯的最终版本下周离职)

2016-04-13 955

版权

本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。

简介： 上次的版本http://blog.chinaunix.net/uid-21142030-id-5613029.html 点击(此处)折叠或打开 [10:58 t ~]$ cat rsync.

上次的版本http://blog.chinaunix.net/uid-21142030-id-5613029.html

点击(此处)折叠或打开

[10:58 t ~]$ cat rsync.t
#!/usr/bin/env bash
#set -x
#2015-08-18: 修复直接压缩201目录的bug
#2015-09-28: 新加项目分类目录,新加多重判断
#2015-10-19: 修复与glusterfs冲突问题.
#2016-04-08: 处理磁盘空紧张时无法处理日志压缩上传导致空间满问题。
#2016-04-11: 解决目录非单独挂载时识别为/无限循环问题。
#2016-04-12: 优化搜索逻辑，添加输出日志。
LOGPATH=$1
PROJECT=$2
SLEEPTIME=1s
DATETIME=$(date +%F_%T)
DATE=$(date +%F)
IP=$(LC_ALL=C /sbin/ifconfig|grep "inet addr:"|grep -v "127.0.0.1"|cut -d: -f2|awk '{print $1}'|sed ':a;$!N;s/\n/_/g;ta;P;D')
LINE=`seq -s= 80|sed 's/[0-9]*//g'`
USER=admin
export RSYNC_PASSWORD=skymobi
RSYNCIP=192.168.1.2
RSYNCMODULE=chss-t
RSYNCCONFMODULE=chss-tc
RSYNCDIR=${PROJECT}/"$IP"
FINDLOGFILE=/tmp/find_"$IP".log.$DATE
RSYNCLOGFILE=/tmp/rsync_"$IP".log.$DATE
RSYNCEXELOGFILE=/tmp/rsync.exe.log.$DATE
REGEX="-regex .*logs/$.*log$?.*201.*$.*log$?$.gz$?"
REGEX2="-regex .*tar.gz"
TIME="-mtime +${DAYS:=1}"
DEL="-exec rm -rfv {} ;"
COMPRESSION="-exec tar -zcvf `basename {}`_${RSYNCMODULE}.tar.gz {} --remove-files --exclude=*tar.gz ;"
#创建目录
echo -e "$LINE\n$DATETIME : Create directory for upload."
mkdir -p /tmp/$RSYNCDIR
echo -e "$DATETIME : Upload create directory.\n$LINE"
for rspath in $RSYNCMODULE $RSYNCCONFMODULE
do
rsync -atv --log-file=${RSYNCLOGFILE} /tmp/${PROJECT} ${USER}@"$RSYNCIP"::${rspath}
done
#扫描日志
for Path in $LOGPATH
do
echo -e "$LINE \n$DATETIME : Operation on the [ $Path ] directory "
#1.是否存在目录
if [ -d /$Path ];then
#########################################
RSYNC(){
if test $# -ne 0;then
rsync -aERtvz --no-p --chmod=Do+rX,Fo-X --log-file=${RSYNCLOGFILE} --remove-source-files --skip-compress=*.gz "$@" ${USER}@"$RSYNCIP"::${ttype:=$RSYNCMODULE}/${RSYNCDIR}
else
echo "$DATETIME : no $RSYNCMODULE.tar.gz file found."
return 1
fi
}
FINDER(){
find -L /${Path} $@
}
SPACECOUNT(){
df -P|grep /${Path} |tail -n 1 |awk '{print $5}'|awk -F% '{print $1}'
}
##########################################
#2.是否空间紧张
echo "$DATETIME : Start while."
starttime=`date`
SPACE=`SPACECOUNT`
Count=0
while test -n "$SPACE" -a "${SPACE:=0}" -gt 20 -a $Count -lt 2
do
if [ $SPACE -gt 80 ];then
echo "$DATETIME : Start del."
TIME="-mmin +720"
FINDER "$REGEX" "-a" "$TIME" "$DEL" >> $FINDLOGFILE
SPACE=`SPACECOUNT`
continue
elif [ $SPACE -gt 60 ];then
echo "$DATETIME : Start upload file direct."
echo '60%
TIME="-mmin +60"
RSYNC `FINDER "$REGEX" "-a" "$TIME"`
SPACE=`SPACECOUNT`
((Count+=1))
continue
#2.
else
echo "$DATETIME : Start compression."
echo 'space
TIME="-mmin +0"
FINDER "$REGEX" "-a ! $REGEX2" "-a" "$TIME" "$COMPRESSION" >> $FINDLOGFILE
SPACE=`SPACECOUNT`
echo "new space is $SPACE"
break
fi
done
endtime=`date`
haldertime=$((`date -d "$endtime" +%s`-`date -d "$starttime" +%s`))
echo "haldertime:$haldertime"
#3.执行上传操作
echo "$DATETIME : Start upload tar.gz file."
starttime=`date`
# for ttype in ${RSYNCMODULE} #${RSYNCCONFMODULE}
# do
RSYNC $(find /$Path -type f -name "*_${RSYNCMODULE}.tar.gz" -a ! -regex '.*/brick/.*')
#RSYNC $(find /$Path -type f -name "*_$ttype.tar.gz" -a ! -regex '.*/brick/.*')
# done
endtime=`date`
haldertime=$((`date -d "$endtime" +%s`-`date -d "$starttime" +%s`))
echo "upload_tar_gz_haldertime:$haldertime"
#1.
else
echo /${Path} does not exist...
fi
echo -e "$DATETIME : [ $Path ] To the end of the directory operation. \n$LINE"
#1.是否存在目录
done >>$RSYNCEXELOGFILE

点击(此处)折叠或打开

[10:58 t ~]$ cat rsync.t
#!/usr/bin/env bash
#set -x
#2015-08-18: 修复直接压缩201目录的bug
#2015-09-28: 新加项目分类目录,新加多重判断
#2015-10-19: 修复与glusterfs冲突问题.
#2016-04-08: 处理磁盘空紧张时无法处理日志压缩上传导致空间满问题。
#2016-04-11: 解决目录非单独挂载时识别为/无限循环问题。
#2016-04-12: 优化搜索逻辑，添加输出日志。
LOGPATH=$1
PROJECT=$2
SLEEPTIME=1s
DATETIME=$(date +%F_%T)
DATE=$(date +%F)
IP=$(LC_ALL=C /sbin/ifconfig|grep "inet addr:"|grep -v "127.0.0.1"|cut -d: -f2|awk '{print $1}'|sed ':a;$!N;s/\n/_/g;ta;P;D')
LINE=`seq -s= 80|sed 's/[0-9]*//g'`
USER=admin
export RSYNC_PASSWORD=skymobi
RSYNCIP=192.168.165.88
RSYNCMODULE=chss-t
RSYNCCONFMODULE=chss-tc
RSYNCDIR=${PROJECT}/"$IP"
FINDLOGFILE=/tmp/find_"$IP".log.$DATE
RSYNCLOGFILE=/tmp/rsync_"$IP".log.$DATE
RSYNCEXELOGFILE=/tmp/rsync.exe.log.$DATE
REGEX="-regex .*logs/$.*log$?.*201.*$.*log$?$.gz$?"
REGEX2="-regex .*tar.gz"
TIME="-mtime +${DAYS:=1}"
DEL="-exec rm -rfv {} ;"
COMPRESSION="-exec tar -zcvf `basename {}`_${RSYNCMODULE}.tar.gz {} --remove-files --exclude=*tar.gz ;"
#创建目录
echo -e "$LINE\n$DATETIME : Create directory for upload."
mkdir -p /tmp/$RSYNCDIR
echo -e "$DATETIME : Upload create directory.\n$LINE"
for rspath in $RSYNCMODULE $RSYNCCONFMODULE
do
rsync -atv --log-file=${RSYNCLOGFILE} /tmp/${PROJECT} ${USER}@"$RSYNCIP"::${rspath}
done
#扫描日志
for Path in $LOGPATH
do
echo -e "$LINE \n$DATETIME : Operation on the [ $Path ] directory "
#1.是否存在目录
if [ -d /$Path ];then
#########################################
RSYNC(){
if test $# -ne 0;then
rsync -aERtvz --no-p --chmod=Do+rX,Fo-X --log-file=${RSYNCLOGFILE} --remove-source-files --skip-compress=*.gz "$@" ${USER}@"$RSYNCIP"::${ttype:=$RSYNCMODULE}/${RSYNCDIR}
else
echo "$DATETIME : no $RSYNCMODULE.tar.gz file found."
return 1
fi
}
FINDER(){
find -L /${Path} $@
}
SPACECOUNT(){
df -P|grep /${Path} |tail -n 1 |awk '{print $5}'|awk -F% '{print $1}'
}
##########################################
#2.是否空间紧张
echo "$DATETIME : Start while."
starttime=`date`
SPACE=`SPACECOUNT`
Count=0
while test -n "$SPACE" -a "${SPACE:=0}" -gt 20 -a $Count -lt 2
do
if [ $SPACE -gt 80 ];then
echo "$DATETIME : Start del."
TIME="-mmin +720"
FINDER "$REGEX" "-a" "$TIME" "$DEL" >> $FINDLOGFILE
SPACE=`SPACECOUNT`
continue
elif [ $SPACE -gt 60 ];then
echo "$DATETIME : Start upload file direct."
echo '60%
TIME="-mmin +60"
RSYNC `FINDER "$REGEX" "-a" "$TIME"`
SPACE=`SPACECOUNT`
((Count+=1))
continue
#2.
else
echo "$DATETIME : Start compression."
echo 'space
TIME="-mmin +0"
FINDER "$REGEX" "-a ! $REGEX2" "-a" "$TIME" "$COMPRESSION" >> $FINDLOGFILE
SPACE=`SPACECOUNT`
echo "new space is $SPACE"
break
fi
done
endtime=`date`
haldertime=$((`date -d "$endtime" +%s`-`date -d "$starttime" +%s`))
echo "haldertime:$haldertime"
#3.执行上传操作
echo "$DATETIME : Start upload tar.gz file."
starttime=`date`
# for ttype in ${RSYNCMODULE} #${RSYNCCONFMODULE}
# do
RSYNC $(find /$Path -type f -name "*_${RSYNCMODULE}.tar.gz" -a ! -regex '.*/brick/.*')
#RSYNC $(find /$Path -type f -name "*_$ttype.tar.gz" -a ! -regex '.*/brick/.*')
# done
endtime=`date`
haldertime=$((`date -d "$endtime" +%s`-`date -d "$starttime" +%s`))
echo "upload_tar_gz_haldertime:$haldertime"
#1.
else
echo /${Path} does not exist...
fi
echo -e "$DATETIME : [ $Path ] To the end of the directory operation. \n$LINE"
#1.是否存在目录
done >>$RSYNCEXELOGFILE

点击(此处)折叠或打开

[11:00 t ~]$ cat while.t
#!/bin/bash
set -x
LOGPATH='opt app data app1'
PROJECT=talentest
sleeptime=0.5h
while :
do
time /bin/bash /bin/rsync.t "$LOGPATH" $PROJECT
sleep $sleeptime
done

巧用shell+rsync实现日志自动过滤处理压缩上传分类(在斯凯的最终版本下周离职)

热门文章

最新文章

相关课程

相关电子书

相关实验场景