ELK日志收集部署

时间同步：

1
2
ntpdate pool.ntp.org
echo  ‘* /5  * * * * ntpdate pool.ntp.org’>>  /var/spool/cron/root

关闭防火墙和selinux

1
2
3
/etc/init .d /iptables   stop
chkconfig iptables off
Sed –I ‘s /SELINUX =enforcing /SELINUX =disabled /g ’   /etc/selinux/config

安装java

1
2
3
4
5
6
7
8
9
10
11
12
13
http: //www .oracle.com /technetwork/java/javase/downloads/jdk8-downloads-2133151 .html
tar  zxf jdk-7u45-linux-x64. tar .gz
mv  jdk1.8.0_92/  /usr/local/jdk
设置jdk环境变量
vi  /etc/profile
-------------------------------------------------------
JAVA_HOME= /usr/local/jdk
PATH=$PATH:$JAVA_HOME /bin
CLASSPATH=.:$JAVA_HOME /lib :$JAVA_HOME /jre/lib
export  JAVA_HOME PATHCLASSPATH
-------------------------------------------------------
source  /etc/profile
java -version


下载redis

1
2
3
4
5
6
7
8
9
10
wget 
http: //download .redis.io /releases/redis-3 .2.3. tar .gz
tar  zxf redis-3.2.3. tar .gz
cd  redis-3.2.3
make 
make  PREFIX= /usr/local/redis  install
mkdir  /usr/local/redis/conf
cp   redis.conf  /usr/local/redis/conf/redis .conf.bak
cd  /usr/local/redis/conf
cp  redis.conf.bak redis.conf

添加环境变量

1
2
3
echo  'PATH=$PATH:/usr/local/redis/bin/'  >> /etc/profilesource  /etc/profile
启动redis：
/usr/local/redis/bin/redis-server  /usr/local/redis/conf/redis .conf &


下载logstash elasticsearch  kibana

1
2
3
4
https: //www .elastic.co /downloads
elasticsearch-5.0.0. tar .gz        
logstash-5.0.0. tar .gz
kibana-5.0.0-linux-x86_64. tar .gz

解压文件：

1
2
3
tar  zxf logstash-5.0.0. tar .gz 
tar  zxf elasticsearch-5.0.0. tar .gz 
tar  zxf kibana-5.0.0-linux-x86_64. tar .gz

移动到统一管理目录：

1
2
3
mv  elasticsearch-5.0.0  /usr/local/elasticsearch
mv  logstash-5.0.0  /usr/local/logstash
mv  kibana-5.0.0-linux-x86_64  /usr/local/kibana


备份配置文件：

1
2
3
cp  /usr/local/logstash/config/logstash .yml  /usr/local/logstash/config/logstash .yml.bak.$( date +%F)      
cp  /usr/local/elasticsearch/config/elasticsearch .yml /usr/local/elasticsearch/config/elasticsearch .yml.bak.$( date  +%F)
cp  /usr/local/kibana/config/kibana .yml  /usr/local/kibana/config/kibana .yml.bak.$( date +%F)


配置Elasticsearch


创建用户

1
2
3
4
默认elasticsearch不支持root用户启动，所以需要先创建一个普通用户
groupadd elastic 
useradd  elastic –g  elastic  –M
chown  -R elastic.elastic    /usr/local/elasticsearch/

修改配置文件：

1
2
3
4
network.host: 192.168.0.248
http.port: 9200
su  elastic
/usr/local/elasticsearch/bin/elasticsearch  –d

验证启动：

curl http://localhost:9200/

添加开机启动：

1
echo  ‘ /usr/local/elasticsearch/bin/elasticsearch-d ’ >> /etc/rc . local

注意错误：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
ERROR: bootstrap checks failed
问题：max  file  descriptors [65535]  for  elasticsearch process likelytoo low, increase to at least [65536]
解决： vi  /etc/security/limits .conf
*             -       nofile          65536
或者
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
问题：max number of threads [1024]  for  user [elasticsearch] likely toolow, increase to at least [2048]
解决： vi /etc/security/limits .d /90-nproc .conf
* soft nproc 2048
需重启生效。
问题：max virtual memory areas vm.max_map_count [65530] likely toolow, increase to at least [262144]
解决： vi  /etc/sysctl .conf
vm.max_map_count=655360
sysctl -p


配置kibana：

1
2
3
4
5
修改配置文件：
vi  /usr/local/kibana/config/kibana .yml
server.port: 5601
server.host:  "192.168.0.248"
elasticsearch.url: http: //192 .168.0.248:9200


测试logstash：

1
2
3
4
cd  logstash-5.0.0
bin /logstash  -e  'input { stdin { } } output {stdout {} }'
hello world
2013-11-21T01:22:14.405+0000 0.0.0.0 helloworld


配置logstash服务端：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
vi  /usr/local/logstash/config/logstash .yml
input {
          redis {
                    host => “192.168.0.248”
                    port => 6379
                    type  =>”redis-input”
                    data_type =>”list”
                    key =>”logstash:redis”
}
}
output{
elasticsearch {
         hosts=> [ "192.168.0.248:9200" ]
            }        
}


客户端：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
vi  /usr/local/logstash/config/logstash .yml
input {
          file {
                    type =>”nginx_access”
                    path=>[” /usr/local/nginx/logs/access .log”]
}
}
output {
          redis{
                    host  => “192.168.0.248”
                    data_type=> “list”
                    key=> “logstash:redis”
}
}


启动客户端：

1
2
3
/usr/local/logstash/bin/logstash  –f  /usr/local/logstash/conf/logstash .conf
或yum安装：
/usr/share/logstash/bin/logstash  -f /etc/logstash/conf .d /logstash .conf

客户端yum安装：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
rpm -- import  https: //artifacts .elastic.co /GPG-KEY-elasticsearch
cat  >  /etc/yum .repos.d /logstash .repo <<EOF
[logstash-5.x]
name=Elastic repository  for  5.x packages
baseurl=https: //artifacts .elastic.co /packages/5 .x /yum
gpgcheck=1
gpgkey=https: //artifacts .elastic.co /GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type =rpm-md
EOF
  
yum clean all
yum  install  logstash –y