一、概述
Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。
Logstash是一个开源的用于收集,分析和存储日志的工具。
Kibana 也是一个开源和免费的工具，Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以汇总、分析和搜索重要数据日志。
Beats是elasticsearch公司开源的一款采集系统监控数据的代理agent，是在被监控服务器上以客户端形式运行的数据收集器的统称，可以直接把数据发送给Elasticsearch或者通过Logstash发送给Elasticsearch，然后进行后续的数据分析活动。Beats由如下组成:
1.Packetbeat：是一个网络数据包分析器，用于监控、收集网络流量信息，
Packetbeat嗅探服务器之间的流量，解析应用层协议，并关联到消息的处理， 其支 持ICMP (v4 and v6)、DNS、HTTP、Mysql、PostgreSQL、Redis、
MongoDB、Memcache等协议；
2. Filebeat：用于监控、收集服务器日志文件，其已取代 logstash forwarder；
3. Metricbeat：可定期获取外部系统的监控指标信息，其可以监控、收集
Apache、HAProxy、MongoDB、MySQL、Nginx、PostgreSQL、
Redis、System、Zookeeper等服务；
4. Winlogbeat：用于监控、收集Windows系统的日志信息；
5. Create your own Beat：自定义beat ，如果上面的指标不能满足需求，elasticsarch鼓励开发者 使用go语言，扩展实现自定义的beats，只需要按照模板，实现监控的输入，日志，输出等即可。
Beats 将搜集到的数据发送到 Logstash，经 Logstash 解析、过滤后，将其发送到 Elasticsearch 存储，并由 Kibana 呈现给用户。
Beats 作为日志搜集器没有Logstash 作为日志搜集器消耗资源，解决了 Logstash 在各服务器节点上占用系统资源高的问题。
Elastic Stack官方下载地址:https://www.elastic.co/downloads。
二、开源实时日志分析系统Elastic Stack 5.0部署:
A.安装依赖包JDK
①关闭防火墙
|
1
2
3
4
|
# cat /etc/selinux/config |grep -v "#"
SELINUX=disabled
##关闭selinux
SELINUXTYPE=targeted
# systemctl stop firewalld ##关闭防火墙
|
②下载安装JDK
|
1
2
3
4
5
6
|
# wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# yum install jre ##这里我只安装的是jre环境，一样可用
# java -version
openjdk version
"1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-b15)
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode
|
B.安装Elasticsearch
①下载安装elasticsearch
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# cat > /etc/yum.repos.d/elasticsearch.repo <<EOF
[elasticsearch-5.x]
name=Elasticsearch repository
for
5.x packages
baseurl=https:
//artifacts
.elastic.co
/packages/5
.x
/yum
gpgcheck=1
gpgkey=https:
//artifacts
.elastic.co
/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type
=rpm-md
EOF
# yum install elasticsearch
# systemctl start elasticsearch
# systemctl status elasticsearch
# /usr/share/elasticsearch/bin/elasticsearch -V ##查看elasticsearch版本
Version: 5.0.1, Build: 080bb47
/2016-11-11T22
:08:49.812Z, JVM: 1.8.0_111
|
注:安装后出现如下问题，致使elasticsearch无法启动
|
1
2
3
4
5
6
7
|
# systemctl status elasticsearch
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one,
then
you should conf...CThreads=N
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error=
'Cannot a ...'
(errno=12)
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2060255232 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid11084.log
|
解决:
配置elasticsearch下的jvm.options：
|
1
2
3
4
5
|
# vi /etc/elasticsearch/jvm.options
-Xms4g
##启用如下两项
-Xmx4g
##-Xms2g ##关闭如下两项
##-Xmx2g
|
②ElasticSearch默认的对外服务的HTTP端口是9200，节点间交互的TCP端口是9300。
|
1
|
# ss -tlnp |grep -E '9200|9300'
|
③测试服务
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
# curl -X GET http://localhost:9200
{
"name"
:
"XVY0Ovb"
,
"cluster_name"
:
"elasticsearch"
,
"cluster_uuid"
:
"tR_H9avzT6Kf4hXWTIfWyA"
,
"version"
: {
"number"
:
"5.0.1"
,
"build_hash"
:
"080bb47"
,
"build_date"
:
"2016-11-11T22:08:49.812Z"
,
"build_snapshot"
:
false
,
"lucene_version"
:
"6.2.1"
},
"tagline"
:
"You Know, for Search"
}
|
也可以使用如下命令测试
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
# curl -i -XGET 'localhost:9200/'
HTTP
/1
.1 200 OK
content-
type
: application
/json
; charset=UTF-8
content-length: 327
{
"name"
:
"XVY0Ovb"
,
"cluster_name"
:
"elasticsearch"
,
"cluster_uuid"
:
"tR_H9avzT6Kf4hXWTIfWyA"
,
"version"
: {
"number"
:
"5.0.1"
,
"build_hash"
:
"080bb47"
,
"build_date"
:
"2016-11-11T22:08:49.812Z"
,
"build_snapshot"
:
false
,
"lucene_version"
:
"6.2.1"
},
"tagline"
:
"You Know, for Search"
}
|
C.安装Logstash
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# cat > /etc/yum.repos.d/logstash.repo <<EOF
[logstash-5.x]
name=Elastic repository
for
5.x packages
baseurl=https:
//artifacts
.elastic.co
/packages/5
.x
/yum
gpgcheck=1
gpgkey=https:
//artifacts
.elastic.co
/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type
=rpm-md
EOF
# yum install logstash
# systemctl start logstash
# systemctl status logstash
# /usr/share/logstash/bin/logstash -V ##查看logstash版本
logstash 5.0.1
|
D.安装Kibana
①安装Kibana
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# cat > /etc/yum.repos.d/kibana.repo <<EOF
[kibana-5.x]
name=Kibana repository
for
5.x packages
baseurl=https:
//artifacts
.elastic.co
/packages/5
.x
/yum
gpgcheck=1
gpgkey=https:
//artifacts
.elastic.co
/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type
=rpm-md
EOF
# yum install kibana
# systemctl start kibana
# systemctl status kibana
# /usr/share/kibana/bin/kibana -V ##查看kibana版本
5.0.1
|
②只需更改如下配置
|
1
2
|
# cat /etc/kibana/kibana.yml |grep -v "#"
server.host:
"192.168.147.128"
|
③检测服务
|
1
|
# ss -tlnp|grep 5601 ##Kibana默认进程名：node ，端口5601
|
E、Beats安装部署
a.安装部署Filebeat
①安装Filebeat
|
1
2
3
4
5
6
|
# curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.0.1-x86_64.rpm
# rpm -ivh filebeat-5.0.1-x86_64.rpm
# systemctl start filebeat
# systemctl status filebeat
# filebeat.sh -version ##查看版本
filebeat version 5.0.1 (amd64), libbeat 5.0.1
|
②配置Filebeat
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
# cd /etc/filebeat
# cp filebeat.yml filebeat.yml.bak
# vi /etc/filebeat/filebeat.yml ##配置filebeat
#============= Filebeat prospectors ===============
filebeat.prospectors:
- input_type: log
paths:
-
/var/log/
*.log
#==================== Outputs =====================
#------------- Elasticsearch output ---------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: [
"localhost:9200"
]
#---------------- Logstash output -----------------
output.logstash:
# The Logstash hosts
hosts: [
"localhost:5043"
]
##只需配置该处，其他默认即可
# filebeat.sh -configtest -e ##验证配置文件
|
③配置Logstash
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
# cat /etc/logstash/conf.d/logstash.conf
input {
beats {
port =>
"5043"
}
}
filter {
grok {
match => {
"message"
=>
"%{COMBINEDAPACHELOG}"
}
}
geoip {
source
=>
"clientip"
}
}
output {
elasticsearch {
hosts => [
"localhost:9200"
]
}
}
# systemctl restart logstash ##重启logstash
# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit ##验证配置文件
Sending Logstash logs to
/var/log/logstash
which
is now configured via log4j2.properties
Configuration OK
# ss -tlnp|grep -E '5043|9600'
|
④配置kibana
浏览器输入http://localhost:5601，配置filebeat的索引（只需输入filebeat-*即可）。
在第一个框里输入filebeat-*后稍等片刻，kibana会自动识别，OK后下面的按钮会由灰色变为可操控的按钮"Create",如上图所示。点击该按钮后，最后就会呈现如下图所示:
我们再回过头新建logstash的索引，浏览器输入http://localhost:5601，点击左边栏的”Management” ===> 然后点击“index Patterns” ===>
===> 然后点击“Add New” ===>
===> 点击“Crete”按钮创建logstash索引，创建完成后即会展现如下图所示:
b.安装部署Packetbeat
①安装配置Packetbeat
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
|
# yum install libpcap
# curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-5.0.1-x86_64.rpm
# rpm -ivh packetbeat-5.0.1-x86_64.rpm
# cat /etc/packetbeat/packetbeat.yml
==================== Network device ===================
packetbeat.interfaces.device: any
##捕获所有消息发送或接收的网络接口
======================== Flows ========================
packetbeat.flows:
timeout: 30s
period: 10s
================== Transaction protocols ==============
##如下是packetbeat默认支持的主要协议及端口
packetbeat.protocols.icmp:
enabled:
true
packetbeat.protocols.amqp:
ports: [5672]
packetbeat.protocols.cassandra:
ports: [9042]
packetbeat.protocols.dns:
ports: [53]
include_authorities:
true
include_additionals:
true
packetbeat.protocols.http:
ports: [80, 8080, 8000, 5000, 8002]
packetbeat.protocols.memcache:
ports: [11211]
packetbeat.protocols.mysql:
ports: [3306]
packetbeat.protocols.pgsql:
ports: [5432]
packetbeat.protocols.redis:
ports: [6379]
packetbeat.protocols.thrift:
ports: [9090]
packetbeat.protocols.mongodb:
ports: [27017]
packetbeat.protocols.nfs:
ports: [2049]
========================= General =========================
========================= Outputs =========================
------------------- Elasticsearch output ------------------
output.elasticsearch:
hosts: [
"localhost:9200"
]
--------------------- Logstash output ---------------------
output.logstash:
hosts: [
"localhost:5043"
]
##只需配置该处，其他默认即可
============================= Logging =====================
|
②验证配置并启动packetbeat
|
1
2
3
4
5
6
7
|
# packetbeat.sh -version ##查看packetbeat版本
packetbeat version 5.0.1 (amd64), libbeat 5.0.1
# packetbeat.sh -configtest -e ##测试配置文件
......
Config OK
# systemctl start packetbeat
# systemctl status packetbeat
|
③配置Kibana(新建packetbeat索引)
在http://localhost:5601下新建索引页面输入“packetbeat-*”，之后kibana会自动更新，在“Time-field name”下面的三个选项中选择“@timestamp”，最后点击“Create”创建即可。
创建完成后，kibana显示如下:
c.安装部署Metricbeat
①安装配置metricbeat
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
# curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.0.1-x86_64.rpm
# rpm -ivh metricbeat-5.0.1-x86_64.rpm
# cat /etc/metricbeat/metricbeat.yml
================= Modules configuration =================
metricbeat.modules:
---------------------- System Module ---------------------
- module: system
metricsets:
- cpu
- load
- filesystem
- fsstat
- memory
- network
- process
enabled:
true
period: 10s
processes: [
'.*'
]
========================= General =======================
========================= Outputs =======================
------------------- Elasticsearch output ----------------
output.elasticsearch:
hosts: [
"localhost:9200"
]
--------------------- Logstash output -------------------
output.logstash:
hosts: [
"localhost:5043"
]
##只需配置该项，其他默认即可
======================= Logging =========================
|
②验证配置并启动metricbeat
|
1
2
3
4
5
|
# metricbeat.sh -version ##查看版本信息
metricbeat version 5.0.1 (amd64), libbeat 5.0.1
# metricbeat.sh -configtest -e ##验证配置文件
# systemctl start metricbeat
# systemctl status metricbeat
|
③配置kibana(新建metricbeat索引)
在http://localhost:5601下新建索引页面输入“metricbeat-*”，之后kibana会自动更新，在“Time-field name”下面的选项中选择“@timestamp”，最后点击“Create”创建即可。
最后呈现如下图所示:
注1:
关于ELK Stack的一些查询语句:
①查询filebeat
|
1
|
# curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'
|
②查询packetbeat
|
1
|
# curl -XGET 'http://localhost:9200/packetbeat-*/_search?pretty'
|
③查询metricbeat
|
1
|
# curl -XGET 'http://localhost:9200/metricbeat-*/_search?pretty'
|
④查询集群健康度
|
1
|
# curl 'localhost:9200/_cat/health?v'
|
⑤查看节点列表
|
1
2
3
|
# curl 'localhost:9200/_cat/nodes?v'
ip heap.percent
ram
.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1 37 93 3 0.05 0.07 0.41 mdi * XVY0Ovb
|
⑥列出所有索引
|
1
2
3
4
5
6
7
8
9
10
11
12
|
# curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow
open
filebeat-2016.11.28 Mn4MzxdTRaCj9iseutcmqA 5 1 2 0 12kb 12kb
yellow
open
filebeat-2016.11.29 iMrr710mT42mApxdV62k-A 5 1 159 0 65.9kb 65.9kb
yellow
open
packetbeat-2016.11.29 wkTcIwD6RgiiCFwlWBIILA 5 1 5652 0 1.6mb 1.6mb
yellow
open
customer NvxXLgHoREefJLRhot13Ug 5 1 0 0 800b 800b
yellow
open
packetbeat-2016.11.28 Beoe07S7QB-dntNV4nxJNQ 5 1 2446 0 676.4kb 676.4kb
yellow
open
test
M7WbkYq2QNmeJ9NOyMfMZA 5 1 0 0 800b 800b
yellow
open
logstash-2016.11.28 pcb_84ChSBe9A7VRd-SQNw 5 1 161 0 123.2kb 123.2kb
yellow
open
metricbeat-2016.11.29 AmVeT1xCQGCnxlAFXUxhYw 5 1 94459 0 37.6mb 37.6mb
yellow
open
logstash-2016.11.29 6PCKMYKCSVmPfdg-Sx2ARA 5 1 85772 0 20.3mb 20.3mb
yellow
open
.kibana QYTg0I5KS-yc3d7GSey3Zw 1 1 5 0 102kb 102kb
|
注2:
如果搭建期间有什么不清楚或不了解的，建议去看官方文档，文档地址如下:
https://www.elastic.co/guide/index.html
注3:如需创建自己的beat，该处有教程可供学习:
http://www.itnose.net/detail/6675459.html
