(一)含义:
logstash是一种分布式日志收集框架,开发语言是JRuby,当然是为了与Java平台对接,不过与Ruby语法兼容良好,非常简洁强大,经常与ElasticSearch,Kibana配置,组成著名的ELK技术栈,非常适合用来做日志数据的分析。
当然它可以单独出现,作为日志收集软件,你可以收集日志到多种存储系统或临时中转系统,如MySQL,redis,kakfa,HDFS, lucene,solr等并不一定是ElasticSearch
(二)安装
1,下载和安装。通过官网进行下载目前最新的版本 https://www.elastic.co/cn/downloads/logstash
1
2
3
4
5
6
7
8
9
10
11
|
[root@localhost
local
]
# wget https://artifacts.elastic.co/downloads/logstash/lo
gstash-5.4.0.
tar
.gz
--2017-05-22 17:05:29-- https:
//artifacts
.elastic.co
/downloads/logstash/logstash-5
.4.0.
tar
.gz
Resolving artifacts.elastic.co... 174.129.40.40, 184.72.234.88, 184.73.227.9, ...
Connecting to artifacts.elastic.co|174.129.40.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 92546670 (88M) [application
/x-gzip
]
Saving to: “logstash-5.4.0.
tar
.gz”
100%[======================================>] 92,546,670 9.10M
/s
in
14s
2017-05-22 17:05:44 (6.52 MB
/s
) - “logstash-5.4.0.
tar
.gz” saved [92546670
/92546670
]
[root@localhost
local
]
# tar xf logstash-5.4.0.tar.gz
|
2,常用的命令
1
2
|
[root@localhost
local
]
# logstash/bin/logstash -V
logstash 2.2.2
|
常用的选项有:
选项:
-f , 指定加载一个后缀为.conf文件的logstash配置模块
-e , 命令行指定参数 , 通常用来调试
-w, 指定logstash的工作线程数
-l, 指定logstash的默认日志写入到一个文件中,如果不指定,默认是标准输出
--quiet 静默模式,仅仅只有error级别信息输出
--verbose info级别的log输出
--debug debug 级别的log输出.
-V, --version 查看logstash的版本
-p, --pluginpath PATH 加载自定义的logstash插件
-t, --configtest 检查logstash配置是否有效
-h, --help 打印帮助
(三)常用的模型
(1)input => output
(2)input => filter => output
具体的解释如下:
a . input常用的输入源有:file ,syslog,redis,log4j,tomcat_log,nging.log,以及各种业务log
b.filter常用的选项有:
grok:支持正则提取任何非结构化数据或结构化数据,其中logstash内置120多种正则,比如常见的时间,ip,用户名,等等也支持自定义正则解析
mutate:修改字段名,删除,更新等操作,转换字段类型等
drop: 删除某些时间,如debug
clone:拷贝一份事件副本,用来添加或删除字段
geoip : 通过ip获取地理位置信息,在做kibana区域统计图非常酷炫
ruby: 支持原生的ruby代码,操作事件,实现强大的其他功能
c.output常用的输出选项有:
elasticsearch 比较常用
file:写入文件
redis:写入队列
hdfs:写入HDFS,需插件支持
zabbix: zabbix监控
mongodb:写入mongodb库
codecs:编码插件,常用来处理json数据或者多行数据源
(四)常见的操作
(1),-e使用命令行调试。结束已ctrl+d直接结束:
1
2
3
4
5
6
7
8
|
[root@localhost logstash]
# /usr/local/logstash/bin/logstash -e "input{stdin{}}
output{stdout{}}"
Settings: Default pipeline workers: 1
Logstash startup completed
hello world
2017-05-22T09:28:04.364Z localhost.localdomain hello world
test
2017-05-22T09:28:52.170Z localhost.localdomain
test
|
(2),-f调试。如果配置比较多的话,可以写成以.conf结尾的配置文件里,然后使用-f 命令进行加载。
1
2
3
4
5
6
7
8
9
10
11
|
[root@localhost etc]
# vim hello.conf
input{stdin{}}
output{stdout{}}
[root@localhost etc]
# /usr/local/logstash/bin/logstash -f hello.conf --configte
st
Configuration OK
[root@localhost etc]
# /usr/local/logstash/bin/logstash -f hello.conf
Settings: Default pipeline workers: 1
Logstash startup completed
hello world
2017-05-22T09:40:23.044Z localhost.localdomain hello world
|
备注:下面是logstash服务脚步
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
|
#! /bin/sh
#
# /etc/rc.d/init.d/logstash
#
# Starts Logstash as a daemon
#
# chkconfig: 2345 90 10
# description: Starts Logstash as a daemon.
### BEGIN INIT INFO
# Provides: logstash
# Required-Start: $local_fs $remote_fs
# Required-Stop: $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: S 0 1 6
# Short-Description: Logstash
# Description: Starts Logstash as a daemon.
### END INIT INFO
.
/etc/rc
.d
/init
.d
/functions
NAME=logstash
DESC=
"Logstash Daemon"
DEFAULT=
/etc/sysconfig/
$NAME
if
[ `
id
-u` -
ne
0 ];
then
echo
"You need root privileges to run this script"
exit
1
fi
# The following variables can be overwritten in $DEFAULT
PATH=
/bin
:
/usr/bin
:
/sbin
:
/usr/sbin
# See contents of file named in $DEFAULT for comments
LS_USER=root
LS_GROUP=root
LS_HOME=
/home/logstash
LS_HEAP_SIZE=
"1500m"
LS_JAVA_OPTS=
"-Djava.io.tmpdir=${LS_HOME}"
LS_LOG_FILE=
/var/log/logstash/
$NAME.log
LS_CONF_DIR=
/etc/logstash/index
.conf
LS_OPEN_FILES=163840
LS_NICE=19
LS_OPTS=
""
LS_PIDFILE=
/var/run/
$NAME.pid
# End of variables that can be overwritten in $DEFAULT
if
[ -f
"$DEFAULT"
];
then
.
"$DEFAULT"
fi
# Define other required variables
PID_FILE=${LS_PIDFILE}
DAEMON=
"/home/logstash/bin/logstash"
DAEMON_OPTS=
"agent -f ${LS_CONF_DIR} -w 18 -l ${LS_LOG_FILE} ${LS_OPTS}"
#
# Function that starts the daemon/service
#
do_start()
{
if
[ -z
"$DAEMON"
];
then
echo
"not found - $DAEMON"
exit
1
fi
if
pidofproc -p
"$PID_FILE"
>
/dev/null
;
then
failure
exit
99
fi
# Prepare environment
HOME=
"${HOME:-$LS_HOME}"
JAVA_OPTS=
"${LS_JAVA_OPTS}"
ulimit
-n ${LS_OPEN_FILES}
cd
"${LS_HOME}"
export
PATH HOME JAVA_OPTS LS_HEAP_SIZE LS_JAVA_OPTS LS_USE_GC_LOGGING
test
-n
"${JAVACMD}"
&&
export
JAVACMD
nice
-n ${LS_NICE} runuser -s
/bin/sh
-c
"exec $DAEMON $DAEMON_OPTS"
${LS_USER} >
/dev/null
1>&1 <
/dev/null
&
RETVAL=$?
local
PID=$!
# runuser forks rather than execing our process.
usleep 500000
JAVA_PID=$(
ps
axo ppid,pid |
awk
-
v
"ppid=$PID"
'$1==ppid {print $2}'
)
PID=${JAVA_PID:-$PID}
echo
$PID > $PID_FILE
[ $PID = $JAVA_PID ] && success
}
#
# Function that stops the daemon/service
#
do_stop()
{
killproc -p $PID_FILE $DAEMON
RETVAL=$?
echo
[ $RETVAL = 0 ] &&
rm
-f ${PID_FILE}
}
case
"$1"
in
start)
echo
-n
"Starting $DESC: "
do_start
touch
/var/run/
$NAME
;;
stop)
echo
-n
"Stopping $DESC: "
do_stop
rm
/var/run/
$NAME
;;
restart|reload)
echo
-n
"Restarting $DESC: "
do_stop
do_start
;;
status)
echo
-n
"$DESC"
status -p $PID_FILE
exit
$?
;;
*)
echo
"Usage: $SCRIPTNAME {start|stop|status|restart}"
>&2
exit
3
;;
esac
echo
exit
0
|