GrayLog多节点集群架构如下
下面介绍Graylog4.2集群部署过程
基础环境准备
三台服务器：CentOS7.9的虚拟机
/data分区（LVM) 用于Elasticsearch日志数据存储
内存大小为6GB
IP地址和主机名如下：
- 192.168.31.211 graylog01 graylog01.walkingcloud.cn
- 192.168.31.212 graylog02 graylog02.walkingcloud.cn
- 192.168.31.213 graylog03 graylog03.walkingcloud.cn
均已关闭SELINUX
一、配置MongoDB集群
1、三台均安装mongodb
cat > /etc/yum.repos.d/mongodb-org.repo << \EOF [mongodb-org] name=MongoDB Repository baseurl=https://mirrors.aliyun.com/mongodb/yum/redhat/$releasever/mongodb-org/4.4/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc EOF yum install -y mongodb-org
2、三台均先启动mongodb
systemctl daemon-reload systemctl enable mongod.service systemctl start mongod.service systemctl --type=service --state=active | grep mongod firewall-cmd --add-port=27017/tcp --permanent --zone=public firewall-cmd --reload
3、主节点生成授权认证keyfile文件并拷贝到其它节点
openssl rand -base64 756 > /var/lib/mongo/access.keyfile chown mongod:mongod /var/lib/mongo/access.keyfile chmod 600 /var/lib/mongo/access.keyfile scp -rp /var/lib/mongo/access.keyfile root@graylog02:/var/lib/mongo/ scp -rp /var/lib/mongo/access.keyfile root@graylog03:/var/lib/mongo/
4、vi修改/etc/mongod.conf
以主节点为例，修改配置文件
vi /etc/mongod.conf net: port: 27017 bindIp: 192.168.31.211 security: keyFile: /var/lib/mongo/access.keyfile replication: replSetName: graylog-rs
graylog02修改成如下
graylog03修改成如下
5、初始化集群
这里为了本地能登录mongo
先将bindIp: 192.168.31.211修改为bindIp: 0.0.0.0 并重启服务 systemctl restart mongod.service
输入mongo进数据库
use admin rs.initiate( { _id : "graylog-rs", members: [ { _id: 0, host: "192.168.31.211:27017" }, { _id: 1, host: "192.168.31.212:27017" }, { _id: 2, host: "192.168.31.213:27017" } ] }) rs.status()查看集群状态
6、创建graylog数据库并设置密码
#修改admin用户密码 use admin db.createUser({user: "admin", pwd: "Admin@2021", roles: ["root"]}) db.auth("admin","Admin@2021") #创建graylog数据库并设置密码 use graylog db.createUser({ user: "graylog", pwd: "Graylog2021", "roles" : [{ "role" : "dbOwner", "db" : "graylog" }, { "role" : "readWrite", "db" : "graylog" }] })
7、将主节点bindIp配置恢复并重启mongod服务
将graylog01的bindIp修改成192.168.31.211并重启服务
这时使用账号和密码登录mongo
在graylog02 重启mongod服务，使graylog01重新变为Primary角色
二、搭建Elasticsearch集群
1、三台节点均以rpm包方式安装Elasticsearch
cd /opt wget https://mirrors.cloud.tencent.com/elasticstack/yum/elastic-7.x/7.16.2/elasticsearch-7.16.2-x86_64.rpm rpm -ivh elasticsearch-7.16.2-x86_64.rpm systemctl daemon-reload systemctl enable elasticsearch.service systemctl start elasticsearch.service firewall-cmd --add-port=9200/tcp --permanent --zone=public firewall-cmd --reload
2、创建存储和日志文件夹
mkdir -p /data/elasticsearch/data mkdir -p /data/elasticsearch/logs chown -R elasticsearch:elasticsearch /data/elasticsearch cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml_default
3、修改elasticsearch.yml配置文件
vi/etc/elasticsearch/elasticsearch.yml
1)graylog01节点
#添加并修改成如下行
cluster.name: graylog-cluster action.auto_create_index: false node.name: graylog01 node.master: true node.data: true path.data: /data/elasticsearch/data path.logs: /data/elasticsearch/logs network.host: 192.168.31.211 http.port: 9200 transport.port: 9300 discovery.seed_hosts: ["192.168.31.211:9300", "192.168.31.212:9300", "192.168.31.213:9300"] cluster.initial_master_nodes: ["graylog01"]
2)graylog02节点
cat /etc/elasticsearch/elasticsearch.yml | grep -v "^#" | grep -v "^$" cluster.name: graylog-cluster action.auto_create_index: false node.name: graylog02 node.master: false node.data: true path.data: /data/elasticsearch/data path.logs: /data/elasticsearch/logs network.host: 192.168.31.212 http.port: 9200 transport.port: 9300 discovery.seed_hosts: ["192.168.31.211:9300", "192.168.31.212:9300", "192.168.31.213:9300"]
3)graylog03节点
cat /etc/elasticsearch/elasticsearch.yml | grep -v "^#" | grep -v "^$" cluster.name: graylog-cluster action.auto_create_index: false node.name: graylog03 node.master: false node.data: true path.data: /data/elasticsearch/data path.logs: /data/elasticsearch/logs network.host: 192.168.31.213 http.port: 9200 transport.port: 9300 discovery.seed_hosts: ["192.168.31.211:9300", "192.168.31.212:9300", "192.168.31.213:9300"]
4、三个节点均修改jvm.options配置文件上内存大小
firewall-cmd --add-port=9300/tcp --permanent --zone=public firewall-cmd --reload vim /etc/elasticsearch/jvm.options 设置jvm内存大小为物理内存的一半 并重启elasticsearch.service systemctl restart elasticsearch.service
5、查看elasticsearch集群状态
curl -s -XGET 'http://192.168.31.211:9200/_cluster/health?pretty=true' curl -s -XGET 'http://192.168.31.211:9200/_cat/nodes?v'
三、安装GraylogServer并配置GraylogServer集群
1、安装jdk，pwgen和graylog-server
yum install -y java-1.8.0-openjdk-headless.x86_64 yum install -y pwgen
rpm -ivh https://packages.graylog2.org/repo/packages/graylog-4.2-repository_latest.rpm yum install graylog-server -y
systemctl enable graylog-server systemctl start graylog-server cp /etc/graylog/server/server.conf /etc/graylog/server/server.conf_default pwgen -N 1 -s 96 XC0Epiv5SnWFdm82nsUWAJN3t2MHaFEaSFHd6RLPf1nzxwnmubT0n7NQdrK8jCDEOS05DtrkGHDDE61490OUJKBOOXIAT4LI echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1 Enter Password: Graylog@2021 10dfabb9595634675701865aa1c6e774d89d59f4a104ab128fbffcdaa3cf8f7b
2、修改graylog的主配置文件server.conf
主节点graylog01的配置如下
/etc/graylog/server/server.conf cat /etc/graylog/server/server.conf | grep -v "^#" | grep -v "^$" is_master = true node_id_file = /etc/graylog/server/node-id password_secret = XC0Epiv5SnWFdm82nsUWAJN3t2MHaFEaSFHd6RLPf1nzxwnmubT0n7NQdrK8jCDEOS05DtrkGHDDE61490OUJKBOOXIAT4LI root_password_sha2 = 10dfabb9595634675701865aa1c6e774d89d59f4a104ab128fbffcdaa3cf8f7b root_timezone = Asia/Shanghai bin_dir = /usr/share/graylog-server/bin data_dir = /var/lib/graylog-server plugin_dir = /usr/share/graylog-server/plugin http_bind_address = 192.168.31.211:9000 http_publish_uri = http://192.168.31.211:9000/ elasticsearch_hosts = http://graylog01:9200,http://graylog02:9200,http://graylog03:9200 rotation_strategy = count elasticsearch_max_docs_per_index = 20000000 elasticsearch_max_number_of_indices = 20 retention_strategy = delete elasticsearch_shards = 4 elasticsearch_replicas = 0 elasticsearch_index_prefix = graylog allow_leading_wildcard_searches = false allow_highlighting = true elasticsearch_analyzer = standard output_batch_size = 500 output_flush_interval = 1 output_fault_count_threshold = 5 output_fault_penalty_seconds = 30 processbuffer_processors = 8 outputbuffer_processors = 16 processor_wait_strategy = blocking ring_size = 65536 inputbuffer_ring_size = 65536 inputbuffer_processors = 2 inputbuffer_wait_strategy = blocking message_journal_enabled = true message_journal_dir = /var/lib/graylog-server/journal lb_recognition_period_seconds = 3 mongodb_uri = mongodb://graylog:Graylog2021@graylog01:27017,graylog02:27017,graylog03:27017/graylog?replicaSet=graylog-rs mongodb_max_connections = 1000 mongodb_threads_allowed_to_block_multiplier = 5 proxied_requests_thread_pool_size = 32
3、重启graylog-server服务，并开放9000 web端口
建议将配置文件scp到graylog02和graylog03上
firewall-cmd --add-port=9000/tcp --permanent --zone=public firewall-cmd --reload systemctl restart graylog-server
graylog02,graylog03节点修改如下三处即可
is_master = true http_bind_address = 192.168.31.213:9000 http_publish_uri = http://192.168.31.213:9000/
登录Graylog主界面查看GrayLog集群情况
1、查看Nodes信息
2、查看Cluster信息
3、添加Input，可以设置成全局的Input
记得三台几点防火墙开放Input设置的端口，例如syslog1514
firewall-cmd --add-port=1514/udp --permanent --zone=public firewall-cmd --reload
Tips：修改graylog的jvm内存大小
本文参考如下链接完成
https://docs.graylog.org/v1/docs/multinode-setup https://docs.mongodb.com/manual/tutorial/deploy-replica-set/ https://docs.mongodb.com/manual/tutorial/deploy-replica-set-with-keyfile-access-control/#std-label-deploy-repl-set-with-auth https://cloud.tencent.com/developer/article/1615815 https://zhuanlan.zhihu.com/p/120698020 https://www.cnblogs.com/opsdemo/p/15035379.html
