AI 助理
备案控制台
开发者社区 开发与运维 文章 正文

3天玩转shell--12.实战编写nginx日志统计脚本【终结篇】

2023-02-16 485
版权
版权声明：
本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《 阿里云开发者社区用户服务协议》和 《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写 侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
本文涉及的产品
日志服务 SLS，月写入数据量 50GB 1个月
简介： 本节课是shell教程系列的最后一篇文章，希望这12篇文章对shell的学习者有帮助。往后有空会发表一些关于golang编程的实战系列文章、或者是k8s相关的文章。

一、功能介绍：

【1】统计各种状态码的数量
【2】统计访问最多的referer
【3】统计访问最高的uri
【4】统计访问最多的 ip和ua
【5】统计每分钟的请求数、流量、请求时间、状态码等

二、通过代码进行讲解

本节是通过编写一段统计访问日志，统计日志的状态码、浏览器类型、URL、流量、状态码分布等信息，这对于网站的维护人员，能很好的掌握网站的运行情况。将附件保存12.example.txt 
#!/bin/bash
#12.sh v1
PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
export LANG=zh_CN.UTF-8
export PATH

#统计状态码并排序
GroupByStatusCode(){
  file=$1
  cat $file |awk -F "|" '{Arry[$5]+=1;total++}END{for(s in Arry){printf "%d\t%.4f\t%s\n",Arry[s],Arry[s]/total,s}}'|sort -nr -k 1
}

#统计IP并排序
GroupByRemoteAddress(){
  file=$1
  cat $file |awk -F "|" '{Arry[$1]+=1;total++}END{for(s in Arry){printf "%d\t%.4f\t%s\n",Arry[s],Arry[s]/total,s}}'|sort -nr -k 1
}

#统计referer并排序
GroupByReferer(){
  file=$1
  cat $file |awk -F "|" '{Arry[$7]+=1;total++}END{for(s in Arry){printf "%d\t%.4f\t%s\n",Arry[s],Arry[s]/total,s}}'|sort -nr -k 1
}

#统计浏览器并排序
GroupByUserAgent(){
  file=$1
  cat $file |awk -F "|" '{Arry[$8]+=1;total++}END{for(s in Arry){printf "%d\t%.4f\t%s\n",Arry[s],Arry[s]/total,s}}'|sort -nr -k 1
}

#统计URi不含参数，并排序
GroupByUri(){
  file=$1
  cat $file |awk -F "|" '{split($4,aa," |?");Arry[aa[3]]+=1;total++}END{for(s in Arry){printf "%d\t%.4f\t%s\n",Arry[s],Arry[s]/total,s}}'|sort -nr -k 1
}

#统计URL并排序
GroupByUrl(){
  file=$1
  cat $file |awk -F "|" '{split($4,aa," ");Arry[aa[2]]+=1;total++}END{for(s in Arry){printf "%d\t%.4f\t%s\n",Arry[s],Arry[s]/total,s}}'|sort -nr -k 1
}

#统计请求方法并排序
GroupByMethod(){
  file=$1
  cat $file |awk -F "|" '{split($4,aa," |?");Arry[aa[2]]+=1;total++}END{for(s in Arry){printf "%s\t%d\t%.4f\n",s,Arry[s],Arry[s]/total}}'|sort -nr -k 2
}

#统计请求时间并排序
GroupByRequestTime(){
  file=$1
  #cat $file |awk -F "|" '{if($14<1){reqs["<1"]++}else if($14>1 && $14<5){reqs["1-5"]++}else if($14>5 && $14<10){reqs["5-10"]++}else{reqs[">10"]++};total++}END{printf "<1\t%.4f\t%d\n1-5\t%.4f\t%d\n5-10\t%.4f\t%d\n>10\t%.4f\t%d\n",reqs["<
1"]/total,reqs["<1"],reqs["1-5"]/total,reqs["1-5"],reqs["5-10"]/total,reqs["5-10"],reqs[">10"]/total,reqs[">10"]}'
  cat $file |awk -F "|" '{
  if($14<1){reqs["<1"]++}
  else if($14>1 && $14<5) {reqs["1-5"]++}
  else if($14>5 && $14<10){reqs["5-10"]++}
  else {reqs[">10"]++};
  total++
  }END{
   printf "<1\t%.4f\t%d\n1-5\t%.4f\t%d\n5-10\t%.4f\t%d\n>10\t%.4f\t%d\n",reqs["<1"]/total,reqs["<1"],reqs["1-5"]/total,reqs["1-5"],reqs["5-10"]/total,reqs["5-10"],reqs[">10"]/total,reqs[">10"]
  }'
}

#统计每分钟请求的数量
TrendTimesCount(){
 file=$1
 cat $file|awk -F "|" '{tms[substr($2,15,5)]+=1}END{for(t in tms)print t" "tms[t]}'
}

#统计每分钟的流量
TrendTimesNetFlow(){
 file=$1
 cat $file|awk -F "|" '{tms[substr($2,15,5)]+=$6}END{for(t in tms)printf "%s\t%.4fMB\n",t,tms[t]/1024/1024}'
}

#统计每分钟的数量、流量、状态码分布、平均请求时间
TrendTimesAll(){
 file=$1
 #cat $file|awk -F "|" 'BEGIN{printf "时间\t数量\t流量[MB]\t请求时间\t20x\t30x\t40x\t50x\t60x\n"}{tm=substr($2,15,5);tms[tm]+=$6;cnt[tm]+=1;reqt[tm]+=$14;if($5 ~ /20/){sc20x[tm]++}else if($5 ~ /30/){sc30x[tm]++}else if($5 ~ /40/){sc40x[
tm]++}else if($5 ~ /50/){sc50x[tm]++}else{sc60x[tm]++};}END{for(t in tms)printf "%s\t%d\t%.4f\t\t%.4f\t\t%d\t%d\t%d\t%d\t%d\n",t,cnt[t],tms[t]/1024/1024,reqt[t]/cnt[t],sc20x[t],sc30x[t],sc40x[t],sc50x[t],sc60x[t]}'
 cat $file|awk -F "|" 'BEGIN{
      printf "时间\t数量\t流量[MB]\t请求时间\t20x\t30x\t40x\t50x\t60x\n"
  }{
      tm=substr($2,15,5);
      tms[tm]+=$6;
      cnt[tm]+=1;
      reqt[tm]+=$14;
      if($5 ~ /20/) {sc20x[tm]++}
      else if($5 ~ /30/) {sc30x[tm]++}
      else if($5 ~ /40/) {sc40x[tm]++}
      else if($5 ~ /50/) {sc50x[tm]++}
      else {sc60x[tm]++};
   }END{
   for(t in tms)
        printf "%s\t%d\t%.4f\t\t%.4f\t\t%d\t%d\t%d\t%d\t%d\n",t,cnt[t],tms[t]/1024/1024,reqt[t]/cnt[t],sc20x[t],sc30x[t],sc40x[t],sc50x[t],sc60x[t]
   }'
}

#帮助函数
Help(){
 echo "Help:"
 echo "\$1 args \$2 logfile"
 echo "s :groupby statuscode"
 echo "h :groupby remoteaddress"
 echo "e :groupby referer"
 echo "a :groupby useragent"
 echo "i :groupby uri"
 echo "r :groupby url"
 echo "t :groupby requestTime"
 echo "m :groupby method"
 echo "tc :trend count request by minute"
 echo "tn :trend count netflow by minute"
 echo "ta :trend count request,netflow,requesTime,statuscode by minute"
 echo "UseAge Example:"
 echo "$0 s access.log"
 echo "$0 h access.log"
 echo "$0 i access.log"
 echo "$0 tn access.log"
 echo "$0 ta access.log"
}

if [ $# -ne 2 ];then
   Help
   exit 1
else
   if [ ! -s $2 ];then
      echo "$2 file does not exist"
      exit 1
   fi
fi

case $1 in
    s|statuscode)
      GroupByStatusCode "$2";;
    h|remoteaddress)
      GroupByRemoteAddress $2;;
    e|referer)
      GroupByReferer $2;;
    a|useragent)
      GroupByUserAgent $2;;
    i|uri)
      GroupByUri $2;;
    r|url)
      GroupByUrl $2;;
    t|requesttime)
      GroupByRequestTime $2;;
    m|method)
      GroupByMethod $2;;
    tc|trendocunt)
      TrendTimesCount $2;;
    tn|trendnet)
      TrendTimesNetFlow $2;;
    ta|trendall)
      TrendTimesAll $2;;
    *)
      Help;;
esac

三.脚本执行结果：

1.打印帮助信息：
help.png
2.统计状态码和IP并排序：
code.png
3.统计请求方法、统计响应时间分布、统计uri
m.png
4.统计每分钟的数量、每分钟的流量、每分钟的状态码走势
t.png

四.附件

113.96.140.246 | [28/May/2020:15:32:55 +0800] | www.shell.com.cn | "GET /pcedu/ios/1106/2448633_1.html HTTP/1.1" | 200 | 8289 | "-" | "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" | "-" | - | - | - | 172.16.239.5 | 0.234 | https  | 47585
172.16.239.156 | [28/May/2020:15:32:55 +0800] | www.shell.com.cn | "GET /product/11fz/1106/intf1224.html HTTP/1.1" | 200 | 1874 | "http://pdlib.shell.com.cnproduct.shell.com.cn" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 1.232 | http  | 35949
172.16.239.158 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /product/11fz/1106/intf1224.html HTTP/1.1" | 200 | 1874 | "http://pdlib.shell.com.cnproduct.shell.com.cn" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.982 | http  | 58399
172.16.246.51 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /jrzcz/jk/wap/1712/intf19897.html HTTP/1.1" | 200 | 2054 | "http://r.shell.com.cnhttp://dl.shell.com.cn/vr/" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.545 | http  | 52384
113.96.140.246 | [28/May/2020:15:32:56 +0800] | office.shell.com.cn | "GET /1282/12825949_3.html HTTP/1.1" | 200 | 31173 | "https://office.shell.com.cn/1282/12825949_2.html" | "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063" | "-" | - | - | - | 172.16.239.5 | 0.645 | https  | 47791
58.55.123.88 | [28/May/2020:15:32:56 +0800] | www1.shell.com.cn | "GET /2015/dl/client/images/btn_client.png HTTP/1.1" | 304 | 0 | "https://js.shell.com/bashTest/2015/dl/css/client.css" | "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" | "-" | - | - | - | 172.16.239.5 | 0.358 | https  | 26877
172.16.239.174 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /jrzcz/jk/pc/1608/intf15725.html HTTP/1.1" | 200 | 1549 | "http://pdlib.shell.com.cnproduct.shell.com.cn" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.435 | http  | 40954
172.16.238.72 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /pcclub/itbbsindex/1702/intf17885.html HTTP/1.1" | 200 | 1766 | "-" | "NodeJs" | "-" | - | - | - | 172.16.239.5 | 0.519 | http  | 55833
172.16.240.137 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /product/fz/2015/1511/intf11701.html HTTP/1.1" | 200 | 0 | "-" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.905 | http  | 46830
113.96.140.245 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 200 | 0 | "https://dl.shell.com.cn/download/409990.html" | "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" | "-" | - | - | - | 172.16.239.5 | 0.715 | https  | 47574
229.137.244.63 | [28/May/2020:15:32:56 +0800] | www1.shell.com.cn | "GET /zt/gz20140905/404/css/404.css HTTP/1.0" | 200 | 4429 | "-" | "-" | "-" | - | - | - | 172.16.239.5 | 0.926 | http  | 43571
113.96.140.245 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 304 | 0 | "https://dl.shell.com.cn/download/385364.html" | "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36 2345Explorer/6.1.0.7934" | "-" | - | - | - | 172.16.239.5 | 0.445 | https  | 47574
113.96.140.245 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 200 | 0 | "https://dl.shell.com.cn/download/385364.html" | "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36 2345Explorer/6.1.0.7934" | "-" | - | - | - | 172.16.239.5 | 0.506 | https  | 48098
113.96.140.245 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 304 | 0 | "https://dl.shell.com.cn/download/63718.html" | "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.68 Safari/537.36 Core/1.53.4033.400 QQBrowser/9.6.12624.400" | "-" | - | - | - | 172.16.239.5 | 0.371 | https  | 48096
113.96.140.245 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 200 | 0 | "https://dl.shell.com.cn/download/63718.html" | "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.68 Safari/537.36 Core/1.53.4033.400 QQBrowser/9.6.12624.400" | "-" | - | - | - | 172.16.239.5 | 0.341 | https  | 48098
113.96.140.245 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 304 | 0 | "https://dl.shell.com.cn/html_2/1/121/id=1051&pn=0.html" | "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" | "-" | - | - | - | 172.16.239.5 | 0.805 | https  | 48098
114.203.12.194 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /xlab/miniapp/2005/intf23253_1.js?t=1590651164841 HTTP/1.1" | 200 | 1243 | "https://servicewechat.com/wx25cb37caf7f44a41/0/page-frame.html" | "Mozilla/5.0 (Linux; Android 8.0.0; Mi Note 2 Build/OPR1.170623.032; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/78.0.3904.62 XWEB/2353 MMWEBSDK/200401 Mobile Safari/537.36 MMWEBID/747 MicroMessenger/7.0.14.1660(0x27000E37) Process/appbrand0 NetType/WIFI Language/zh_CN ABI/arm64 WeChat/arm64" | "-" | - | - | - | 172.16.239.5 | 0.998 | https  | 44620
172.16.238.228 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /jrzcz/jk/wap/1712/intf19897.html HTTP/1.1" | 200 | 2054 | "http://r.shell.com.cnhttp://dl.shell.com.cn/vr/" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.486 | http  | 50761
110.80.141.159 | [28/May/2020:15:32:56 +0800] | www.shell.com.cn | "GET /pcedu/soft/wl/brower/0311/242394.html HTTP/1.1" | 302 | 270 | "-" | "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" | "-" | - | - | - | 172.16.239.5 | 0.015 | https  | 38277
113.96.140.246 | [28/May/2020:15:32:56 +0800] | pcedu.shell.com.cn | "GET /1346/13462969.html HTTP/1.1" | 200 | 11872 | "https://www.baidu.com/link?url=4cyuuadApwd6GD33cnmpoToHdfj9zL1lQ4mJSsE43ROVRFUJ43OyJziAxJK8KPBMkYefrsIZr3CObtpXvHRG2q&wd=&eqid=c14c6a850005b8910.6600065ecf6924" | "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36" | "-" | - | - | - | 172.16.239.5 | 2.265 | https  | 49769
113.96.140.246 | [28/May/2020:15:32:56 +0800] | mobile.shell.com.cn | "GET /380/3806282.html HTTP/1.1" | 200 | 28100 | "-" | "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" | "-" | - | - | - | 172.16.239.5 | 0.181 | https  | 31225
172.16.238.146 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /jrzcz/jk/wap/1712/intf19897.html HTTP/1.1" | 200 | 2054 | "http://r.shell.com.cnhttp://dl.shell.com.cn/vr/" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.705 | http  | 53098
113.96.140.246 | [28/May/2020:15:32:57 +0800] | gps.shell.com.cn | "GET /553/5535909.html HTTP/1.1" | 200 | 13072 | "-" | "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)" | "-" | - | - | - | 172.16.239.5 | 0.255 | https  | 49801
113.96.140.245 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 200 | 0 | "https://dl.shell.com.cn/download/1655767.html" | "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363" | "-" | - | - | - | 172.16.239.5 | 0.304 | https  | 49684
172.16.239.59 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /product/11fz/1106/intf1224.html HTTP/1.1" | 200 | 1874 | "http://pdlib.shell.com.cnproduct.shell.com.cn" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.905 | http  | 35067
110.80.141.158 | [28/May/2020:15:32:57 +0800] | pcedu.shell.com.cn | "GET /1239/12390087.html HTTP/1.1" | 200 | 14525 | "https://www.so.com/link?m=a92lOp6pOJvedmxUx9TBeQMp4n8OlvOlrcFo88bNTlcmQHQGUIPkwTIJIGjfOGAQByXA1tYZ5GHFd7DO0SYwzXg%2Bv5NwTD1QSYf2AYR4GfD9H3jXpaj9nv5Az8Yf%2BKkCohL2N9RZGto6%2B5coh%2BJvERvajHydHfQ4YTH59HnxjqG6lwQIelamKmPceG3YQCP0LvFYjgwLSCLuER8G%2FmtJwk1RLZ6rpaNq5NlFtXw%3D%3D" | "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko" | "-" | - | - | - | 172.16.239.5 | 0.941 | https  | 36322
150.138.109.205 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /2014/0121/zt4201891.html HTTP/1.1" | 301 | 290 | "-" | "Mozilla/5.0 (Linux; Android 7.0; FRD-AL00 Build/HUAWEIFRD-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/53.0.2785.49 Mobile MQQBrowser/6.2 TBS/043602 Safari/537.36 MicroMessenger/6.5.16.1120 NetType/WIFI Language/zh_CN" | "-" | - | - | - | 172.16.239.5 | 0.387 | http  | 59501
172.16.239.174 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /product/11fz/1106/intf1224.html HTTP/1.1" | 200 | 1874 | "http://pdlib.shell.com.cnproduct.shell.com.cn" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.515 | http  | 41052
172.16.246.51 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /jrzcz/jk/wap/1712/intf19897.html HTTP/1.1" | 200 | 2054 | "http://r.shell.com.cnhttp://dl.shell.com.cn/vr/" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.968 | http  | 52389
172.16.239.175 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /jrzcz/jk/pc/1608/intf15725.html HTTP/1.1" | 200 | 1549 | "http://pdlib.shell.com.cnproduct.shell.com.cn" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.311 | http  | 60251
113.96.140.245 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 304 | 0 | "https://dl.shell.com.cn/download/359460-1.html" | "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" | "-" | - | - | - | 172.16.239.5 | 0.781 | https  | 49684
112.90.135.106 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 200 | 0 | "https://dl.shell.com.cn/download/2352841.html" | "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko" | "-" | - | - | - | 172.16.239.5 | 0.878 | https  | 38873
172.16.239.22 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /pdbk/itbk/jd/ac/1712/10519871.html HTTP/1.0" | 200 | 27162 | "-" | "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" | "116.179.32.71, 172.16.237.58, 172.16.238.216, 172.16.238.216" | - | - | - | 172.16.239.5 | 0.214 | http  | 10246
172.16.237.200 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /pdbk/itbk/software/excel/1403/4509076.html HTTP/1.0" | 200 | 8078 | "https://www.so.com/link?m=as5SxBFhuuzOANq9bvfy5AcMPf8wMcuiS45Td%2FDwcSHQ744TpvL%2BM3iHG7CDeRYSptnK6efNNN3zWfp3mg4hDRhiUD%2BY0ss8pxOZ9tP04Xo2NGclBG9JVuB1PCXzHD1wiVOf5OUT%2F6tSjGvszO4fhPn3CHTXY%2FFRVXMHlyWGjzwsiWkkNChoxzJwhN64cseN5XBAUP9aMrrfVqFj4GtMqmToYKYc1RFwY8bkGncb1oFj4qedamsvLKGfEy6r0cdRS8y%2BZZDOjngsAZKhYw1rkUztOwVCFrdA1BtgsRC%2FuMdwjnLD6IhB1qepAnk4qZrbQ" | "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" | "113.65.30.180" | - | - | - | 172.16.239.5 | 0.199 | http  | 43433
27.128.146.74 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 304 | 0 | "https://dl.shell.com.cn/download/2380882.html" | "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" | "-" | - | - | - | 172.16.239.5 | 0.048 | https  | 47253
27.148.141.118 | [28/May/2020:15:32:57 +0800] | hk.shell.com.cn | "GET //network.shell.com.cn/1344/13441764.html HTTP/1.1" | 302 | 270 | "-" | "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Media Center PC 4.0; SLCC1; .NET CLR 3.0.04320)" | "-" | - | - | - | 172.16.239.5 | 0.590 | http  | 40505
172.16.237.200 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /pdbk/itbk/software/dnyw/1707/9686422.html HTTP/1.0" | 200 | 8420 | "-" | "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362" | "171.113.81.245, 222.79.64.76" | - | - | - | 172.16.239.5 | 0.634 | http  | 43442
27.148.141.119 | [28/May/2020:15:32:57 +0800] | mobile.shell.com.cn | "GET /453/4537978.html HTTP/1.1" | 200 | 13638 | "-" | "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" | "-" | - | - | - | 172.16.239.5 | 0.538 | https  | 62475
172.16.239.54 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /product/11fz/1106/intf1224.html HTTP/1.1" | 200 | 1874 | "http://pdlib.shell.com.cnproduct.shell.com.cn" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.670 | http  | 55058
27.128.146.76 | [28/May/2020:15:32:57 +0800] | pcedu.shell.com.cn | "GET /1055/10554305.html HTTP/1.1" | 200 | 17286 | "https://www.baidu.com/link?url=QkXeaFixVvfQKX6LyyE29rGja5mQwx_L8_ehdHNnl_NO5bJ_TAlul2OpKpen2IJQOp-bXieCVidwkjP2o2HGjK&wd=&eqid=f67b455a0003bf310.8340045ecf6926" | "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36" | "-" | - | - | - | 172.16.239.5 | 2.265 | https  | 47929
27.128.146.76 | [28/May/2020:15:32:57 +0800] | www.shell.com.cn | "GET /factory_pic/883/8830500_pic.html?imgsrc=//img0.shell.com.cn/bashTest/1702/14/8830500_1.jpg&channel=309 HTTP/1.1" | 200 | 9638 | "-" | "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" | "-" | - | - | - | 172.16.239.5 | 0.206 | https  | 47997
112.90.135.106 | [28/May/2020:15:32:58 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 304 | 0 | "https://dl.shell.com.cn/download/1825853.html" | "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6788.400 QQBrowser/10.3.2776.400" | "-" | - | - | - | 172.16.239.5 | 0.220 | https  | 38873
27.148.141.117 | [28/May/2020:15:32:58 +0800] | tv.shell.com.cn | "GET /1283/12830092.html HTTP/1.1" | 200 | 32856 | "https://product.shell.com.cn/lcd_tv/sony/1156127.html" | "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36" | "-" | - | - | - | 172.16.239.5 | 0.635 | https  | 37993
112.90.135.106 | [28/May/2020:15:32:58 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 200 | 0 | "https://dl.shell.com.cn/download/1825853.html" | "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6788.400 QQBrowser/10.3.2776.400" | "-" | - | - | - | 172.16.239.5 | 0.058 | https  | 39865
124.236.26.106 | [28/May/2020:15:32:58 +0800] | www.shell.com.cn | "GET /projector/437/4375785.html HTTP/1.1" | 200 | 14556 | "-" | "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot)" | "-" | - | - | - | 172.16.239.5 | 0.119 | https  | 62999
112.90.135.106 | [28/May/2020:15:32:58 +0800] | www.shell.com.cn | "GET /jrzcz/activitys/2016/1611/intf17385.js HTTP/1.1" | 200 | 0 | "https://dl.shell.com.cn/download/2684489.html" | "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3947.100 Safari/537.36" | "-" | - | - | - | 172.16.239.5 | 0.286 | https  | 39865
172.16.238.61 | [28/May/2020:15:32:58 +0800] | www.shell.com.cn | "GET /global/navibar_html/index.html HTTP/1.1" | 200 | 13418 | "-" | "NodeJs" | "-" | - | - | - | 172.16.239.5 | 0.182 | http  | 50513
172.16.236.29 | [28/May/2020:15:32:58 +0800] | picture.shell.com.cn | "GET /diy/1112/2619852_6.html HTTP/1.0" | 200 | 4446 | "-" | "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)" | "46.229.168.143, 172.16.237.171, 172.16.237.171, 172.16.239.12" | - | - | - | 172.16.239.5 | 0.321 | http  | 33494
172.16.239.59 | [28/May/2020:15:32:58 +0800] | www.shell.com.cn | "GET /product/11fz/1106/intf1224.html HTTP/1.1" | 200 | 1874 | "http://pdlib.shell.com.cnproduct.shell.com.cn" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.683 | http  | 35075
172.16.239.158 | [28/May/2020:15:32:58 +0800] | www.shell.com.cn | "GET /jrzcz/jk/pc/1608/intf15725.html HTTP/1.1" | 200 | 1549 | "http://pdlib.shell.com.cnproduct.shell.com.cn" | "cn.bashTest.JavaHttpClient" | "-" | - | - | - | 172.16.239.5 | 0.898 | http  | 58592
文章标签：
日志服务
Go
Shell
容器
应用服务中间件
Kubernetes
nginx
关键词：
日志服务统计
实战日志服务
Shell脚本
Nginx实战
日志服务实战
相关实践学习
日志服务之使用Nginx模式采集日志
本文介绍如何通过日志服务控制台创建Nginx模式的Logtail配置快速采集Nginx日志并进行多维度分析。
运维包工头
目录
相关文章
瓴羊Dataphin
|
3天前
|
存储 运维 监控
API明细日志及运维统计日志全面提升API可运维性
在数字化转型的大潮中，数据已成为企业最宝贵的资产之一。而数据服务API可快速为数据应用提供数据接口。面对越来越多的API以及越来越多的应用调用，如何快速查看API的服务情况、异常情况及影响范围，以及查看API的调用详情，进行API的性能优化、错误排查变得越来越重要，本文将介绍如何配置和开通API运维统计及明细日志，以及如何查看日志进行介绍。
瓴羊Dataphin
26 0
小王老师呀
|
3天前
|
Java Maven Spring
超实用的SpringAOP实战之日志记录
【11月更文挑战第11天】本文介绍了如何使用 Spring AOP 实现日志记录功能。首先概述了日志记录的重要性及 Spring AOP 的优势，然后详细讲解了搭建 Spring AOP 环境、定义日志切面、优化日志内容和格式的方法，最后通过测试验证日志记录功能的准确性和完整性。通过这些步骤，可以有效提升系统的可维护性和可追踪性。
小王老师呀
13 1
Codelinghu
|
1月前
|
Java 程序员 应用服务中间件
「测试线排查的一些经验-中篇」&& 调试日志实战
「测试线排查的一些经验-中篇」&& 调试日志实战
Codelinghu
20 1
「测试线排查的一些经验-中篇」&& 调试日志实战
白雾茫茫丶
|
3月前
|
SQL 运维 监控
Nest.js 实战 (十)：使用 winston 打印和收集日志记录
这篇文章介绍了在Nest服务中如何使用Winston记录日志。文章首先强调了日志记录在后台服务中的重要性，接着提到Nest默认的内部日志记录器，并指出可以通过@nestjs/common包中的Logger类来全面控制日志系统的行为。文章还提到，为了在生产环境中实现更高级的日志功能，可以使用如Winston之类的Node.js日志包。接下来，文章介绍了如何在Nest服务中使用Winston记录日志，包括安装相关依赖、创建winston配置文件以及实现简单的日志记录示例。最后，文章指出更高级的自定义日志功能需要读者自己去探索。
白雾茫茫丶
118 2
Nest.js 实战 (十)：使用 winston 打印和收集日志记录
lsug6eziqmdfk1111
|
1月前
|
监控 网络协议 CDN
阿里云国际监控查询流量、用量查询流量与日志统计流量有差异？
阿里云国际监控查询流量、用量查询流量与日志统计流量有差异？
lsug6eziqmdfk1111
37 0
yuanzhengme
|
3月前
|
应用服务中间件 nginx
nginx error日志 client intended to send too large body: 1434541 bytes 如何处理？
【8月更文挑战第27天】nginx error日志 client intended to send too large body: 1434541 bytes 如何处理？
yuanzhengme
271 6
游客mldfis24krfue
|
3月前
|
应用服务中间件 Linux nginx
在Linux中，如何统计ip访问情况？分析 nginx 访问日志？如何找出访问页面数量在前十位的ip？
在Linux中，如何统计ip访问情况？分析 nginx 访问日志？如何找出访问页面数量在前十位的ip？
游客mldfis24krfue
125 2
蓝易云
|
2月前
|
SQL 安全 数据库
基于SQL Server事务日志的数据库恢复技术及实战代码详解
基于事务日志的数据库恢复技术是SQL Server中一个非常强大的功能，它能够帮助数据库管理员在数据丢失或损坏的情况下，有效地恢复数据。通过定期备份数据库和事务日志，并在需要时按照正确的步骤恢复，可以最大限度地减少数据丢失的风险。需要注意的是，恢复数据是一个需要谨慎操作的过程，建议在执行恢复操作之前，详细了解相关的操作步骤和注意事项，以确保数据的安全和完整。
蓝易云
115 0
土木林森
|
3月前
|
数据库 Java 监控
Struts 2 日志管理化身神秘魔法师，洞察应用运行乾坤，演绎奇幻篇章！
【8月更文挑战第31天】在软件开发中，了解应用运行状况至关重要。日志管理作为 Struts 2 应用的关键组件，记录着每个动作和决策，如同监控摄像头，帮助我们迅速定位问题、分析性能和使用情况，为优化提供依据。Struts 2 支持多种日志框架（如 Log4j、Logback），便于配置日志级别、格式和输出位置。通过在 Action 类中添加日志记录，我们能在开发过程中获取详细信息，及时发现并解决问题。合理配置日志不仅有助于调试，还能分析用户行为，提升应用性能和稳定性。
土木林森
55 0
土木林森
|
3月前
|
SQL 安全 测试技术
【数据守护者必备】SQL数据备份与恢复策略全解析：从全量到日志备份，手把手教你确保企业信息万无一失的实战技巧！
【8月更文挑战第31天】数据库是企业核心业务数据的基石，为防止硬件故障、软件错误或人为失误导致的数据丢失，制定可靠的备份与恢复策略至关重要。本文通过一个在线购物平台的案例，详细介绍了使用 SQL Server 进行全量备份、差异备份及事务日志备份的方法，并演示了如何利用 SQL Server Agent 实现自动化备份任务。此外，还提供了数据恢复的具体步骤和测试建议，确保数据安全与业务连续性。
土木林森
145 0

热门文章

最新文章

  • 1
    如何优雅打印nginx header和body
  • 2
    nginx反代varnish缓存服务器实现后端amp动静分离架构
  • 3
    [译]Nginx和Lua
  • 4
    安装nginx：src/os/unix/ngx_user.c:26:7: error: ‘struct crypt_data’ has no member named ‘curren
  • 5
    一款爽翻了的Nginx可视化配置平台
  • 6
    nagios监控nginx status
  • 7
    nginx跳转
  • 8
    nginx HTTP Upstream模块
  • 9
    nginx+keepalived构建主主负载均衡代理服务器
  • 10
    nginx禁止ip访问web
  • 1
    编程日志01:个人网站更新用户头像
    50
  • 2
    编程日记02：个人站优化数据库和日志
    46
  • 3
    关于日志的清理
    60
  • 4
    Spring日志框架
    42
  • 5
    日志解析神器——Logstash中的Grok过滤器使用详解
    518
  • 6
    一文详解K8s环境下Job类日志采集方案
    173
  • 7
    logcat 只打印符合包名的log
    51
  • 8
    JNI Log 日志输出
    122
  • 9
    聊聊日志硬扫描，阿里 Log Scan 的设计与实践
    350
  • 10
    【Docker专栏】Docker日志管理与监控的最佳方法
    897

    • 相关课程

    更多
  • 日志服务SLS实现云产品可观测
  • 日志服务 SLS 可观测数据分析平台介绍
  • 大数据知识图谱系列—基于ELK+Flink日志全观测最佳实践
  • Nginx企业级Web服务实战
  • Linux Web服务器Nginx搭建与配置
  • 场景实践-基于阿里云Quick BI 对MOOC网站日志分析

    • 相关电子书

    更多
  • 2019大数据技术公开课第五季—日志数据如何同步到MaxCompute之实践讲解
  • CentOS Nginx PHP JAVA 多语言镜像使用手
  • CentOS Nginx PHP JAVA多语言镜像使用手册

    • 相关实验场景

    更多
  • 通过日志服务实现云资源OSS的安全审计
  • 如何将OSS数据导入至SLS中进行分析
  • 日志服务之使用Nginx模式采集日志
  • Arthas中help、history、keymap、options基础命令应用
  • Arthas中cat 、cls、echo、grep基础命令应用
  • Alibaba Cloud Linux操作系统Shell程序
    • 下一篇
    阿里云OSS设置跨域访问