创建文档生成索引
[elk@es-master ~]$ curl -H "Content-Type:application/json" -XPUT 'http://192.168.25.130:9200/index_name/type_name/1?pretty' -d '{ "name": "xuwl", "age": 18, "job": "Linux" }' { "_index" : "index_name", "_type" : "index_type", "_id" : "1", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 0, "_primary_term" : 1 } - `-H` '指定内容类型 - `-X` '指定http请求方式，这里为PUT上传方式 - `http://192.168.25.130:9200` '指定一台es服务器对外的http端口 - `/index_name` '文档的索引名称,必须小写 - `/type_name` '文档的类型名称,必须小写 - `/1` '文档的ID编号 - `?pretty` '人性化创建索引 - `-d` '指定使用JSON方式来撰写上传文档 - `{ "name": "xuwl", "age": 18, "job": "Linux" }'` '使用JSON格式来撰写上传文档内容
[elk@es-master ~]$ curl -XGET 'http:#192.168.25.130:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open index_name uK-E0UPMTamByd24eamfUQ 1 1 1 0 8.3kb 4.1kb [root@els-master ~]# curl -XGET 'http://192.168.25.130:9200/_cat/shards?v' index shard prirep state docs store ip node index_name 0 p STARTED 1 4.1kb 192.168.25.130 es-master index_name 0 r STARTED 1 4.2kb 192.168.25.128 es-node2
部署redis
'redis-beat 安装配置' [root@redis-beat ~]# yum install gcc gcc-c++ -y [root@redis-beat ~]# cd /usr/local [root@redis-beat local]# wget http://download.redis.io/releases/redis-5.0.5.tar.gz [root@redis-beat local]# tar zxvf redis-5.0.5.tar.gz [root@redis-beat local]# cd redis-5.0.5 [root@redis-beat redis-5.0.5]# make [root@redis-beat redis-5.0.5]# make install [root@redis-beat ~]# ./utils/install_server.sh # 启动redis，配置redis
[root@redis-beat ~]# systemctl daemon-reload [root@redis-beat ~]# systemctl start redis_6379 [root@redis-beat ~]# systemctl status redis_6379.service [root@redis-beat ~]# /sbin/chkconfig redis_6379 on # 设置开机启动 [root@redis-beat ~]# vim /etc/redis/6379.conf # 修改成如下内容 bind 0.0.0.0 protected-mode no # 重启Redis [root@redis-beat ~]# systemctl status redis_6379.service
# 测试 [root@redis-beat ~]# netstat -nltp|grep redis [root@redis-beat ~]# ./redis-cli 127.0.0.1:6379> ping PONG
部署head
# 安装nodejs [root@els-master ~]# curl --silent --location https://rpm.nodesource.com/setup_10.x | sudo bash [root@els-master ~]# yum install -y nodejs # 验证 [root@els-master ~]# node -v v10.16.0 [root@els-master ~]# npm -v 6.9.0 # 配置node源为taobao源 [root@els-master ~]# npm config set registry https://registry.npm.taobao.org
# 安装head插件 [root@els-master ~]# git clone git://github.com/mobz/elasticsearch-head.git [root@els-master ~]# cd elasticsearch-head [root@els-master ~]# npm install --registry=https://registry.npm.taobao.org [root@els-master ~]# npm run start & [1] 15239 [elk@els-master elasticsearch-head]$ > elasticsearch-head@0.0.0 start /home/elk/elasticsearch-head > grunt server Running "connect:server" (connect) task Waiting forever... Started connect web server on http://localhost:9100
部署cerebro（es监控）
# 创建用户 [root@redis-beat ~]# sudo useradd -s /sbin/nologin cerebro
[root@redis-beat ~]# wget https://github.com/lmenezes/cerebro/releases/download/v0.9.2/cerebro-0.9.2.tgz [root@redis-beat ~]# mkdir /opt/cerebro [root@redis-beat ~]# tar xf cerebro-0.9.2.tgz -C /opt/cerebro [root@redis-beat ~]# ln -s /opt/cerebro/cerebro-0.9.2 /opt/cerebro/current [root@redis-beat ~]# chown -R cerebro.cerebro /opt/cerebro
# 更改配置 [root@redis-beat ~]# mkdir /home/cerebro/data [root@redis-beat ~]# chown -R cerebro.cerebro /home/cerebro [root@redis-beat ~]# mv /opt/cerebro/current/conf/application.conf{,.bak} [root@redis-beat ~]# tee /opt/cerebro/current/conf/application.conf << 'EOF' secret="ki:s:[[@=Ag?QI`W2jMwkY:eqvrJ]JqoJyi2axj3ZvOv^/KavOT4ViJSv?6YY4[N" basePath="/" pidfile.path="/opt/cerebro/current/cerebro.pid" data.path="/home/cerebro/data/cerebro.db" es={ gzip=true } auth={ # 访问Cerebro的用户名及密码 type: basic settings: { username="admin" password="1234.com" } } hosts=[ # 要监控的Elasticsearch集群，host:节点访问地址，name:标识，一般用ES的cluster_name { host="http://192.168.152.137:9200" name="es_log" } ] EOF
# 启动cerebro [root@redis-beat ~]# /opt/cerebro/cerebro-0.9.2/bin/cerebro [info] play.api.Play - Application started (Prod) [info] p.c.s.AkkaHttpServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
# 创建systemctl管理启动 [root@redis-beat ~]# vim /opt/cerebro/current/bin/cerebro JAVA_HOME=/usr/local/java # 否则会找不到变量，systemctl启动不了cerebro [root@redis-beat ~]# tee /etc/systemd/system/cerebro.service << 'EOF' [Unit] Description=Cerebro After=network.target [Service] Type=folking PIDFile=/opt/cerebro/current/cerebro.pid User=cerebro Group=cerebro LimitNOFILE=65535 ExecStart=/opt/cerebro/current/bin/cerebro -Dconfig.file=/opt/cerebro/current/conf/application.conf Restart=on-failure WorkingDirectory=/opt/cerebro/current [Install] WantedBy=multi-user.target EOF [root@redis-beat ~]# systemctl daemon-reload [root@redis-beat ~]# systemctl enable cerebro [root@redis-beat ~]# systemctl start cerebro [root@redis-beat ~]# systemctl status cerebro
部署logstash
[elk@logstash ~]$ sudo mkdir /usr/local/elkapp && sudo mkdir -p /usr/local/elkdata/logstash/{date,logs} && sudo chown -R elk.elk /usr/local/elk* [elk@logstash local]$ tar xf logstash-7.8.0.tar.gz -C /usr/local/elkapp/ [elk@logstash local]$ cd elkapp/ [elk@logstash elkapp]$ ln -s logstash-7.8.0/ logstash [elk@logstash elkapp]$ sudo chown elk.elk /usr/local/elk* -R
# 配置logstash.yml [elk@logstash elkapp]$ sudo vim logstash-7.8.0/config/logstash.yml path.data: /usr/local/elkdata/logstash/data path.logs: /usr/local/elkdata/logstash/logs # 配置piplines.yml [elk@logstash logstash]$ sudo mkdir conf.d [elk@logstash logstash]$ vim config/pipelines.yml - pipeline.id: test pipeline.workers: 1 path.config: "/usr/local/elkapp/logstash/conf.d/*.conf"
# 启动logstash [elk@logstash ~]$ /usr/local/elkapp/logstash/bin/logstash -f config/input-output.conf -t - -f 指定配置文件 - -t 检查配置是否语法正确
配置systemctl管理启动
[elk@logstash ~]$ sudo vim /usr/local/elkapp/logstash/bin/logstash JAVA_HOME=/usr/local/java [elk@logstash ~]$ sudo vim /etc/systemd/system/logstash.service # 内容如下 [Unit] Description=logstash [Service] User=elk Group=elk LimitMEMLOCK=infinity LimitNOFILE=100000 LimitNPROC=100000 ExecStart=/usr/local/elkapp/logstash/bin/logstash [Install] WantedBy=multi-user.target [elk@logstash ~]$ sudo systemctl daemon-reload [elk@logstash ~]$ sudo systemctl start logstash [elk@logstash ~]$ sudo systemctl enable logstash
部署filebeat
tar包安装
# 下载安装包 [root@redis-beat ~]# cd /usr/local/src [root@redis-beat src]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.0-linux-x86_64.tar.gz [root@redis-beat src]# tar -xf filebeat-7.8.0-linux-x86_64.tar.gz -C /usr/local/ [root@redis-beat src]# cd /usr/local/filebeat-7.8.0-linux-x86_64 [root@redis-beat local]# ln -s filebeat-7.8.0-linux-x86_64/ filebeat [root@redis-beat filebeat-7.8.0-linux-x86_64]# ./filebeat -e -c filebeat.yml # 测试
# 配置systemctl启动 [root@redis-beat ~]# tee /etc/systemd/system/filebeat.service << 'EOF' [Unit] Description=filebeat server daemon Documentation= /usr/local/filebeat/filebeat -help Wants=network-online.target After=network-online.target [Service] User=root Group=root Environment="BEAT_CONFIG_OPTS=-c /usr/local/filebeat/filebeat.yml" ExecStart= /usr/local/filebeat/filebeat $BEAT_CONFIG_OPTS Restart=always [Install] WantedBy=multi-user.target EOF [root@redis-beat ~]# systemctl daemon-reload # 重载system 配置 [root@redis-beat ~]# systemctl start filebeat # 启动filebeat服务 [root@redis-beat ~]# systemctl enable filebeat # 设置开机自启动 [root@redis-beat ~]# systemctl disable filebeat # 停止开机自启动 [root@redis-beat ~]# systemctl status filebeat # 查看服务当前状态 [root@redis-beat ~]# systemctl restart filebeat # 重新启动服务 [root@redis-beat ~]# systemctl list-units --type=service # 查看所有已启动的服务
yum安装
[root@redis-beat ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch [root@redis-beat ~]# tee /etc/yum.repos.d/elastic.repo << 'EOF' # 内容如下 [elastic-7.x] name=Elastic repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF [root@redis-beat ~]# yum install filebeat
rpm 安装
[root@redis-beat ~]# curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.0-x86_64.rpm [root@redis-beat ~]# rpm -vi filebeat-7.8.0-x86_64.rpm
Ubuntu-apt-get
# 导入签名Key [elk@beat ~]$ sudo wget -qO - https://#artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - [elk@beat ~]$ sudo apt-get install apt-transport-https # 保存仓库到/etc/apt/sources.list.d/elastic-7.x.list [elk@beat ~]$ echo "deb https://#artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list [elk@beat ~]$ sudo apt-get update && sudo apt-get install filebeat # 配置文件在/etc/filebeat/filebeat.yml # 配置filebeat开机启动 [elk@beat ~]$ sudo update-rc.d filebeat defaults 95 10
Ubuntu-deb
[elk@beat ~]$ curl -L -O https://#artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.0-amd64.deb
Docker
[root@redis-beat ~]# docker pull docker.elastic.co/beats/filebeat:7.8.0 [root@redis-beat ~]# docker run \ docker.elastic.co/beats/filebeat:7.8.0 \ setup -E setup.kibana.host=kibana:5601 \ -E output.elasticsearch.hosts=["elasticsearch:9200"]
配置Filebeat
# 配置文件filebeat.yml。 可以通过以下命令创建日志文件，并进行内容追加，以便进行写入测试 [root@redis-beat ~]# touch /usr/local/access-filebeat-test.log [root@redis-beat ~]# echo "this msg is from reids" >> /usr/local/access-filebeat-test.log
部署kibana
[root@es-node2-kibana ~]# su - elk # 创建elkaPP目录并设置所有者 [elk@es-node2-kibana ~]$ sudo mkdir /usr/local/elkapp # 创建ELK数据目录并设置所有者 [elk@es-node2-kibana ~]$ sudo mkdir /usr/local/elkdata # 创建logstash主目录 [elk@es-node2-kibana ~]$ sudo mkdir -p /usr/local/elkdata/kibana # 创建logstash数据目录 [elk@es-node2-kibana ~]$ sudo mkdir -p /usr/local/elkdata/kibana/data # 创建logstash日志目录 [elk@es-node2-kibana ~]$ sudo mkdir -p /usr/local/elkdata/kibana/logs # 设置目录权限 [elk@es-node2-kibana ~]$ sudo chown -R elk:elk /usr/local/elkapp [elk@es-node2-kibana ~]$ sudo chown -R elk:elk /usr/local/elkdata ##### 合并版命令 ##### [elk@es-node2-kibana ~]$ sudo mkdir /usr/local/elkapp && sudo mkdir -p /usr/local/elkdata/kibana/{data,logs} && sudo chown -R elk:elk /usr/local/elkapp && sudo chown -R elk:elk /usr/local/elkdata
# 安装kibana [elk@es-node2-kibana ~]$ cd /usr/local/src [elk@es-node2-kibana src]$ sudo wget https://artifacts.elastic.co/downloads/kibana/kibana-7.8.0-linux-x86_64.tar.gz [elk@es-node2-kibana src]$ sudo tar -xf kibana-7.8.0-linux-x86_64.tar.gz -C /usr/local/elkapp [elk@es-node2-kibana src]$ cd /usr/local/elkapp [elk@es-node2-kibana local]$ sudo ln -s kibana-7.8.0-linux-x86_64 kibana # 设置目录权限 [elk@es-node2-kibana local]$ sudo chown -R elk:elk /usr/local/elkapp [elk@es-node2-kibana local]$ sudo chown -R elk:elk /usr/local/elkdata ##### 合并版命令 ##### [elk@es-node2-kibana src]$ sudo tar -zxvf kibana-7.8.0-linux-x86_64.tar.gz -C /usr/local/elkapp && sudo ln -s kibana-7.8.0-linux-x86_64 kibana && sudo chown -R elk:elk /usr/local/elkapp && sudo chown -R elk:elk /usr/local/elkdata
# 配置kibana [elk@es-node2-kibana ~]$ vim /usr/local/elkapp/kibana-7.8.0-linux-x86_64/config/kibana.yml # 增加如下内容 server.port: 5601 server.host: "192.168.152.128" elasticsearch.hosts: ["http:#192.168.152.130:9200"]
# 配置systemctl启动 [elk@es-node2-kibana ~]$ vi /etc/systemd/system/kibana.service # 内容如下 [Unit] Description=kibana [Service] User=elk Group=elk LimitMEMLOCK=infinity LimitNOFILE=100000 LimitNPROC=100000 ExecStart=/usr/local/elkapp/kibana/bin/kibana [Install] WantedBy=multi-user.target [elk@es-node2-kibana ~]$ sudo systemctl daemon-reload [elk@es-node2-kibana ~]$ sudo systemctl start kibana [elk@es-node2-kibana ~]$ sudo systemctl enable kibana ##### 合并版命令 ##### [elk@es-node2-kibana ~]$ sudo systemctl daemon-reload && sudo systemctl start kibana && sudo systemctl enable kibana
# 汉化kibana [elk@es-node2-kibana ~]vim /usr/local/elkapp/kibana/config/kibana.yaml i18n.locale: "zh-CN"
