【大数据开发运维解决方案】Elasticsearch+Logstash+Kibana(6.7.1版本)安装部署

2023-03-24 149

版权

本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。

本文涉及的产品

检索分析服务 Elasticsearch 版，2核4GB开发者规格 1个月

云原生大数据计算服务 MaxCompute，5000CU*H 100GB 3个月

云原生大数据计算服务MaxCompute，500CU*H 100GB 3个月

简介： Elasticsearch+Logstash+Kibana(6.7.1版本)安装部署目前Elasticsearch、Logstash、Kibana三个组件都准备安装在虚拟机供个人学习使用。一、部署Elasticsearch1、下载安装包官网下载地址：ES下载官网选择Elasticsearch组件2、上传解压安装包[root@s133061 elk]# pwd/hadoop/elk[root@s133061 elk]# lselasticsearch-6.7.1.tar.gz kiba

Elasticsearch+Logstash+Kibana(6.7.1版本)安装部署

目前Elasticsearch、Logstash、Kibana三个组件都准备安装在虚拟机供个人学习使用。

一、部署Elasticsearch

1、下载安装包

官网下载地址：
ES下载官网
选择Elasticsearch组件

2、新增ES用户

因为elasticsearch不允许root用户启动，所以需要创建新的用户和组。


[root@s133061 elasticsearch-6.7.1]# useradd es
[root@s133061 elasticsearch-6.7.1]# passwd es
Changing password for user es.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.

3、上传解压安装包

[root@s133061 elk]# pwd
/hadoop/elk
[root@s133061 elk]# ls
elasticsearch-6.7.1.tar.gz  kibana-6.7.1-linux-x86_64.tar.gz  logstash-6.7.1.tar.gz

[root@s133061 elk]# tar -xvf elasticsearch-6.7.1.tar.gz -C /home/es/

4、#修改elasticsearch的权限给es用户和组

[root@s133061 elk]# cd /home/es/
[root@s133061 es]# ls
elasticsearch-6.7.1
[root@s133061 es]# chown -R es:es elasticsearch-6.7.1
[root@s133061 es]# ll
total 0
drwxr-xr-x 8 es es 143 Apr  3  2019 elasticsearch-6.7.1

5、修改配置文件

[root@s133061 es]# cd elasticsearch-6.7.1/
[root@s133061 elasticsearch-6.7.1]# ls
bin  config  lib  LICENSE.txt  logs  modules  NOTICE.txt  plugins  README.textile
[root@s133061 elasticsearch-6.7.1]# cd config/
[root@s133061 config]# ls
elasticsearch.yml  jvm.options  log4j2.properties  role_mapping.yml  roles.yml  users  users_roles
切换用户为es后创建目录：
先创建es的日志目录和数据存放目录：
[es@s133061 elasticsearch-6.7.1]$ mkdir -p data/es
[es@s133061 elasticsearch-6.7.1]$ mkdir -p data/logs/es
接下来修改配置文件：
[es@s133061 elasticsearch-6.7.1]$ pwd
/home/es/elasticsearch-6.7.1
[es@s133061 elasticsearch-6.7.1]$ cd config/
[es@s133061 config]$ vim elasticsearch.yml
修改部分配置如下：
path.data: /home/es/elasticsearch-6.7.1/data/es
path.logs: /home/es/elasticsearch-6.7.1/data/logs/es
--目的是使ES支持跨域请求
network.host: 10.241.133.61
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: '*'

6、修改linux系统资源配置

下面是本人在上面配置修改后启动报的错误以及解决方案，建议各位先按照下面配置修改自己配置再启动es。

a.max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
修改 /etc/security/limits.conf 文件，增加配置，来改变用户 es 每个进程最大同时打开文件数的大小：
es soft nofile 65535 
es hard nofile 65537
可切换到es用户下，然后通过下面2个命令查看当前数量：
ulimit -Hn
ulimit -Sn
注意：用户退出重新登录后配置才会刷新生效。

b. max number of threads [3818] for user [es] is too low, increase to at least [4096]
最大线程个数太低。修改配置文件 /etc/security/limits.conf ，增加配置：
es - nproc 4096 
# 或者 
es soft nproc 4096 
es hard nproc 4096
可切换到es用户下，然后通过下面2个命令查看当前最大线程数：
ulimit -Hu
ulimit –Su
注意：用户退出重新登录后配置才会刷新生效。
c. max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
修改 /etc/sysctl.conf 文件，在文末增加配置
vm.max_map_count=262144
执行命令sysctl -p生效。
d. memory locking requested for elasticsearch process but memory is not locked
修改 /etc/security/limits.conf 文件，增加配置：
* soft memlock unlimited 
* hard memlock unlimited

7、启动elasticsearch

[es@s133061 elasticsearch-6.7.1]$ cd bin/
[es@s133061 bin]$ nohup ./elasticsearch &
[1] 23324
[es@s133061 bin]$ nohup: ignoring input and appending output to ‘nohup.out’

访问url:http://10.241.133.61:9200/
出现下面提示代表es启动成功：

{
  "name" : "GRQFxQq",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "j916An9yQ9GSJ6eA5qt_-w",
  "version" : {
    "number" : "6.7.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "2f32220",
    "build_date" : "2019-04-02T15:59:27.961366Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

8、停止命令

netstat -ntlp | grep 9200 | awk '{print $7}' | awk -F '/' '{print $1}' | xargs kill -9

二、Logstash部署

1、下载安装包

官网下载地址：
ES下载官网
选择logstash组件

2、解压缩

[root@s133061 elk]# pwd
/hadoop/elk
[root@s133061 elk]# tar -xvf logstash-6.7.1.tar.gz
[root@s133061 elk]# ls
elasticsearch-6.7.1.tar.gz  kibana-6.7.1-linux-x86_64.tar.gz  logstash-6.7.1  logstash-6.7.1.tar.gz
[root@s133061 elk]# cd logstash-6.7.1/
[root@s133061 logstash-6.7.1]# ls
bin  config  CONTRIBUTORS  data  Gemfile  Gemfile.lock  lib  LICENSE.txt  logstash-core  logstash-core-plugin-api  modules  NOTICE.TXT  tools  vendor  x-pack
[root@s133061 logstash-6.7.1]# mkdir zhaoyd
[root@s133061 logstash-6.7.1]# cd zhaoyd

3、在ES创建索引

创建索引
[root@s133061 zhaoyd]#  curl -XPUT http://10.241.133.61:9200/zydtest
{"acknowledged":true,"shards_acknowledged":true,"index":"zydtest"}
查询刚刚创建的索引：
[root@s133061 zhaoyd]# curl -XGET "http://10.241.133.61:9200/zydtest/_search?pretty"
{
  "took" : 85,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}

4、Logstash向Elasticsearch导数测试

编辑测试数据：

[root@s133061 zhaoyd]# cat log.log
2020-12-14T10:50:58.000+00:00 INFO this is a test! 10.241.133.13

编辑配置文件：

[root@s133061 zhaoyd]# cat test.conf
input{
        file{
                path => "/hadoop/elk/logstash-6.7.1/zhaoyd/log.log"
                start_position => "beginning"
        }
}
filter{
        grok{
                match=>{ "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{DATA:message} %{IP:address}" }
        }
}
output{
        elasticsearch{
                hosts => "10.241.133.61:9200"
                index => "zydtest"
        }
        stdout {}
}

执行导入任务，将log.log文件内容导入到ES：

[root@s133061 logstash-6.7.1]# pwd
/hadoop/elk/logstash-6.7.1
[root@s133061 logstash-6.7.1]# ./bin/logstash -f zhaoyd/test.conf
Sending Logstash logs to /hadoop/elk/logstash-6.7.1/logs which is now configured via log4j2.properties
[2020-12-16T16:36:34,297][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-12-16T16:36:34,319][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.7.1"}
[2020-12-16T16:36:44,171][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2020-12-16T16:36:44,792][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.241.133.61:9200/]}}
[2020-12-16T16:36:45,063][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.241.133.61:9200/"}
[2020-12-16T16:36:45,147][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2020-12-16T16:36:45,152][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2020-12-16T16:36:45,195][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.241.133.61:9200"]}
[2020-12-16T16:36:45,217][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2020-12-16T16:36:45,315][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2020-12-16T16:36:45,429][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2020-12-16T16:36:46,042][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/hadoop/elk/logstash-6.7.1/data/plugins/inputs/file/.sincedb_a1dc436238800e668fd2df0a021e5d89", :path=>["/hadoop/elk/logstash-6.7.1/zhaoyd/log.log"]}
[2020-12-16T16:36:46,106][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1fe96bd7 run>"}
[2020-12-16T16:36:46,197][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2020-12-16T16:36:46,204][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-12-16T16:36:46,834][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
/hadoop/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
       "message" => [
        [0] "2020-12-14T10:50:58.000+00:00 INFO this is a test! 10.241.133.13",
        [1] "this is a test!"
    ],
    "@timestamp" => 2020-12-16T08:36:47.058Z,
          "host" => "s133061",
     "log-level" => "INFO",
       "address" => "10.241.133.13",
      "@version" => "1",
     "timestamp" => "2020-12-14T10:50:58.000+00:00",
          "path" => "/hadoop/elk/logstash-6.7.1/zhaoyd/log.log"
}

上面导入任务已经执行成功了，接下来查询下索引数据：

[root@s133061 logstash-6.7.1]# curl -XGET "http://10.241.133.61:9200/zydtest/_search?pretty"
{
  "took" : 25,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "zydtest",
        "_type" : "doc",
        "_id" : "ZR6xanYBLtVoEIHt0n-X",
        "_score" : 1.0,
        "_source" : {
          "message" : [
            "2020-12-14T10:50:58.000+00:00 INFO this is a test! 10.241.133.13",
            "this is a test!"
          ],
          "@timestamp" : "2020-12-16T08:36:47.058Z",
          "host" : "s133061",
          "log-level" : "INFO",
          "address" : "10.241.133.13",
          "@version" : "1",
          "timestamp" : "2020-12-14T10:50:58.000+00:00",
          "path" : "/hadoop/elk/logstash-6.7.1/zhaoyd/log.log"
        }
      }
    ]
  }
}

数据已经能够正常导入了，证明安装没问题。

三、安装部署Kibana

1、下载安装包

官网下载地址：
ES下载官网
选择Kibana组件

2、解压

[root@s133061 elk]# pwd
/hadoop/elk
[root@s133061 elk]# tar -xvf kibana-6.7.1-linux-x86_64.tar.gz
[root@s133061 elk]# mv kibana-6.7.1-linux-x86_64 kibana-6.7.1

3、配置参数文件

[root@s133061 elk]# cd kibana-6.7.1/
[root@s133061 kibana-6.7.1]# ls
bin  built_assets  config  data  LICENSE.txt  node  node_modules  NOTICE.txt  optimize  package.json  plugins  README.txt  src  target  webpackShims
[root@s133061 kibana-6.7.1]# cd config/
[root@s133061 config]# vim kibana.yml
#解除注释，并修改成以下内容
server.port: 5601
server.host: "10.241.133.61"     #允许远程访问
elasticsearch.hosts: ["http://10.241.133.61:9200"]    //修改成自己集群的地址
kibana.index: ".kibana"

4、启动Kibana

[root@s133061 bin]# pwd
/hadoop/elk/kibana-6.7.1/bin
[root@s133061 bin]# nohup ./kibana &

【大数据开发运维解决方案】Elasticsearch+Logstash+Kibana(6.7.1版本)安装部署

Elasticsearch+Logstash+Kibana(6.7.1版本)安装部署

一、部署Elasticsearch

1、下载安装包

2、新增ES用户

3、上传解压安装包

4、#修改elasticsearch的权限给es用户和组

5、修改配置文件

6、修改linux系统资源配置

7、启动elasticsearch

二、Logstash部署

1、下载安装包

2、解压缩

3、在ES创建索引

4、Logstash向Elasticsearch导数测试

三、安装部署Kibana

1、下载安装包

2、解压

3、配置参数文件

热门文章

最新文章

相关课程

相关电子书

相关实验场景

热门

活动广场

任务中心

开发者评测

高校计划

乘风者计划

训练营

阿里云MVP

话题

直播

下载

镜像站

技术资料

插件

【大数据开发运维解决方案】Elasticsearch+Logstash+Kibana(6.7.1版本)安装部署

Elasticsearch+Logstash+Kibana(6.7.1版本)安装部署

一、部署Elasticsearch

1、下载安装包

2、新增ES用户

3、上传解压安装包

4、#修改elasticsearch的权限给es用户和组

5、修改配置文件

6、修改linux系统资源配置

7、启动elasticsearch

二、Logstash部署

1、下载安装包

2、解压缩

3、在ES创建索引

4、Logstash向Elasticsearch导数测试

三、安装部署Kibana

1、下载安装包

2、解压

3、配置参数文件

热门文章

最新文章

相关课程

相关电子书

相关实验场景