拓扑环境：
实施要求：
1.所有群集禁止关闭防火墙，需手动开放端口
2.搭建LVS服务器群集，负载web群集
3.配置keepalived高可用服务，实现LVS高可用
4.搭建web服务器群集（apache），配置相同
5.在web服务器上搭建Logstash来获取日志信息，传递到内网EK服务器
6.搭建Elasticsearch+Kibana服务器，来查看web服务的日志信息
7.搭建GFS储存服务器（分布式复制卷），给web服务器提供网站信息，支持冗余
8.搭建zabbix监控服务，监控web服务器的资源占用情况，出现问题即使处理
9.外网所有的服务器为zabbix管理服务器做密钥对验证，方便统一管理
注：如有忽略地方 请实施部门自行添加
保证整体服务高可用，高负载，高安全性！！！
实验服务器分配：
centos7-1-LVS（DR）-keepalived：202.202.2.1
centos7-2-LVS（DR）-keepalived：202.202.2.2
centos7-3-web-logstash：192.168.3.1
centos7-4-web-logstash：192.168.3.2
centos7-5-GFS：192.168.3.3
centos7-6-GFS：192.168.3.4
centos7-7-Elasticsearch-Kibana：192.168.3.5
centos7-8-zabbix：202.202.2.3,192.168.3.6
一，部署两台keepalived服务器群集
将两台LVS服务的IP地址配置完成
[root@localhost ~]# ifconfig ens33 ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 202.202.2.1 netmask 255.255.255.0 broadcast 202.202.2.255 inet6 fe80::20c:29ff:fe15:e99e prefixlen 64 scopeid 0x20<link> ether 00:0c:29:15:e9:9e txqueuelen 1000 (Ethernet) RX packets 1488 bytes 227261 (221.9 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 845 bytes 128422 (125.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@localhost ~]# ifconfig ens33 ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 202.202.2.2 netmask 255.255.255.0 broadcast 202.202.2.255 inet6 fe80::20c:29ff:fe95:af58 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:95:af:58 txqueuelen 1000 (Ethernet) RX packets 720 bytes 155002 (151.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 378 bytes 63798 (62.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
部署（主）keepalived部署（从）keepalived，配置不同（路由器的名称，优先级，以及热备状态不同）
[root@localhost ~]# yum -y install keepalived ipvsadm [root@localhost ~]# cd /etc/keepalived/ [root@localhost keepalived]# cp keepalived.conf keepalived.conf.bak [root@localhost keepalived]# vim keepalived.conf global_defs { router_id LVS_HA_R1 } vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 202.202.2.254 } } [root@localhost keepalived]# systemctl start keepalived
部署（从）keepalived，配置不同（路由器的名称，优先级，以及热备状态不同）
[root@localhost network-scripts]# cd /etc/keepalived/ [root@localhost keepalived]# cp keepalived.conf keepalived.conf.bak [root@localhost keepalived]# vi keepalived.conf global_defs { router_id LVS_HA_R2 } vrrp_instance VI_1 { state BACKUP interface ens33 virtual_router_id 51 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 202.202.2.254 } } [root@localhost keepalived]# systemctl start keepalived
调整/proc的参数（关闭内核的重定向参数）
[root@localhost network-scripts]# vi /etc/sysctl.conf [root@localhost network-scripts]# sysctl -p net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.ens33.send_redirects = 0
二，部署两台web（apache）群集
配置两台的IP地址，内网和外网，双网卡
[root@localhost ~]# ifconfig ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 202.202.2.3 netmask 255.255.255.0 broadcast 202.202.2.255 inet6 fe80::20c:29ff:fe1c:a1a4 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:1c:a1:a4 txqueuelen 1000 (Ethernet) RX packets 777 bytes 163737 (159.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 278 bytes 36258 (35.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ens36: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.3.1 netmask 255.255.255.0 broadcast 192.168.3.255 inet6 fe80::20c:29ff:fe1c:a1ae prefixlen 64 scopeid 0x20<link> ether 00:0c:29:1c:a1:ae txqueuelen 1000 (Ethernet) RX packets 243 bytes 33270 (32.4 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 168 bytes 28988 (28.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@localhost ~]# ifconfig ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 202.202.2.4 netmask 255.255.255.0 broadcast 202.202.2.255 inet6 fe80::20c:29ff:feb5:b978 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:b5:b9:78 txqueuelen 1000 (Ethernet) RX packets 1468 bytes 242924 (237.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 827 bytes 86606 (84.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ens36: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.3.2 netmask 255.255.255.0 broadcast 192.168.3.255 inet6 fe80::20c:29ff:feb5:b982 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:b5:b9:82 txqueuelen 1000 (Ethernet) RX packets 157 bytes 19848 (19.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 178 bytes 34316 (33.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
安装apache（两台相同），开放80端口
1. [root@localhost ~]# yum -y install httpd 2. [root@localhost ~]# firewalld-cmd --add-port=80/tcp
三，配置LVS+keepalived高可用群集
编辑web服务器池的配置，两台配置相同
[root@localhost ~]# vim /etc/keepalived/keepalived.conf virtual_server 202.202.2.254 80 { #虚拟机服务器地址（VIP） delay_loop 6 lb_algo rr lb_kind DR protocol TCP real_server 202.202.2.3 80 { #第一个web节点的地址和端口 weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 4 } } real_server 202.202.2.4 80 { #第二个web节点的地址和端口 weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 4 } } } [root@localhost ~]# systemctl restart keepalived
配置web，两台配置相同
[root@localhost ~]# cd /etc/sysconfig/network-scripts/ [root@localhost network-scripts]# cp ifcfg-lo ifcfg-lo:0 [root@localhost network-scripts]# vi ifcfg-lo:0 DEVICE=lo:0 IPADDR=202.202.2.254 NETMASK=255.255.255.255 ONBOOT=yes NAME=loopback:0 [root@localhost network-scripts]# ifup lo:0 [root@localhost network-scripts]# route add -host 202.202.2.254 dev lo:
[root@localhost network-scripts]# sysctl -p net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.default.arp_ignore = 1 net.ipv4.conf.default.arp_announce = 2 net.ipv4.conf.lo.arp_ignore = 1 net.ipv4.conf.lo.arp_announce = 2
现在已经实现了LVS(DR) 负载均衡和高可用了。
之后配置GFS为web服务器提供统一的web文件
四，配置GFS群集为web服务器提供文件
注：为GFS服务器各添加两块硬盘，大小根据实际情况，我以实验为目的各给两个G
配置初始化环境
1. [root@localhost ~]# vi /etc/hosts 2. 192.168.3.3 gfs-01 3. 192.168.3.4 gfs-02
为每一台GFS服务器创建主分区，格式化，挂载分区（两台相同）
[root@localhost ~]# fdisk /dev/sdb [root@localhost ~]# fdisk /dev/sdc [root@localhost ~]# mkfs.xfs /dev/sdb1 [root@localhost ~]# mkfs.xfs /dev/sdc1 [root@localhost ~]# mkdir -p /www/html-01 [root@localhost ~]# mkdir -p /www/html-02 [root@localhost ~]# mount /dev/sdb1 /www/html-01 [root@localhost ~]# mount /dev/sdc1 /www/html-02 [root@localhost ~]# vi /etc/fstab /dev/sdb1 /www/html-01 xfs default 0 0 /dev/sdc1 /www/html-02 xfs default 0 0
安装glusfs软件包
[root@localhost ~]# yum -y install glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma
启动GlusterFS
[root@localhost ~]# systemctl start glusterd [root@localhost ~]# systemctl enable glusterd Created symlink from /etc/systemd/system/multi-user.target.wants/glusterd.service to /usr/lib/systemd/system/glusterd.service.
添加节点，在gfs-01上添加gfs-02节点
[root@localhost ~]# gluster peer probe gfs-01 peer probe: success. Probe on localhost not needed [root@localhost ~]# gluster peer probe gfs-02 peer probe: success.
常见分布式复制卷
[root@localhost ~]# gluster volume create dis-rep replica 2 gfs-01:/www/html-01 gfs-01:/www/html-02 gfs-02:/www/html-01 gfs-02:/www/html-02 force volume create: dis-rep: success: please start the volume to access data [root@localhost ~]# gluster volume start dis-rep volume start: dis-rep: success
在两台web服务器上安装Gluster客户端，挂载dis-rep卷
[root@localhost network-scripts]# yum -y install glusterfs glusterfs-fuse [root@localhost ~]# vim /etc/hosts 192.168.3.3 gfs-01 192.168.3.4 gfs-02 [root@localhost network-scripts]# mount -t glusterfs gfs-01:dis-rep /var/www/html/ [root@localhost network-scripts]# vi /etc/fstab gfs-01:dis-rep /var/www/html glusterfs default,_netdev 0 0
到此，web服务器的存储高可用完成。
五，配置Elasticsearch+Kibana服务器
配置EK服务器的地址
[root@localhost ~]# ifconfig ens33 ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.3.5 netmask 255.255.255.0 broadcast 192.168.3.255 inet6 fe80::20c:29ff:fe38:64dc prefixlen 64 scopeid 0x20<link> ether 00:0c:29:38:64:dc txqueuelen 1000 (Ethernet) RX packets 1591 bytes 241540 (235.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1024 bytes 129928 (126.8 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
安装Elasticsearch软件
[root@localhost ~]# rpm -ihv /mnt/elasticsearch-5.5.0.rpm [root@localhost ~]# systemctl daemon-reload [root@localhost ~]# systemctl enable elasticsearch.service Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
更改Elasticsearch主配置文件
node.name: node-1 path.data: /data/elk_data path.logs: /var/log/elasticsearch bootstrap.memory_lock: false network.host: 0.0.0.0 http.port: 9200
创建数据存放路径并授权
1. [root@localhost ~]# mkdir -p /data/elk_data 2. [root@localhost ~]# chown elasticsearch:elasticsearch /data/elk_data
启动elasticsearch并查看是否成功开启
[root@node1 ~]# systemctl start elasticsearch [root@node1 ~]# netstat -natp | grep 9200 tcp6 0 0 :::9200 :::* LISTEN 11437/java [root@node1 ~]#
安装kibana程序
[root@node1 ~]# rpm -ihv /mnt/kibana-5.5.1-x86_64.rpm 警告：/mnt/kibana-5.5.1-x86_64.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY 准备中... ################################# [100%] 正在升级/安装... 1:kibana-5.5.1-1 ################################# [100%] [root@node1 ~]# systemctl enable kibana
设置kibana的主配置文件/etc/kibana/kibana.yml
server.port: 5601 server.host: "0.0.0.0" elasticsearch.url: "http://192.168.3.5:9200" kibana.index: ".kibana"
启动kibana服务
[root@node1 ~]# systemctl start kibana [root@node1 ~]# netstat -antp | grep 5601 tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 11586/node
六，在web服务器上部署logstash服务
安装logstash程序,两台web服务器配置相同
[root@localhost ~]# rpm -ihv /mnt/logstash-5.5.1.rpm 警告：/mnt/logstash-5.5.1.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY 准备中... ################################# [100%] 正在升级/安装... 1:logstash-1:5.5.1-1 ################################# [100%] Using provided startup.options file: /etc/logstash/startup.options Successfully created system startup script for Logstash [root@localhost ~]# systemctl start logstash
编写logstash配置文件apache_log.conf
[root@localhost ~]# cd /etc/logstash/conf.d/ [root@localhost conf.d]# vi apache_log.conf input { file{ path => "/etc/httpd/logs/access_log" type => "access" start_position => "beginning" } file{ path => "/etc/httpd/logs/error_log" type => "error" start_position => "beginning" } } output { if [type] == "access" { elasticsearch { hosts => ["192.168.3.5:9200"] index => "apache_access-%{+YYYY.MM,dd}" } } if [type] == "error" { elasticsearch { hosts => ["192.168.3.5:9200"] index => "apache_error-%{+YYYY.MM,dd}" } } } [root@localhost conf.d]# /usr/share/logstash/bin/logstash -f apache_log.conf [root@localhost conf.d]# systemctl restart logstash
登录Kibana，添加索引查看日志
七，安装zabbix服务器，监控LVS和Web服务器的状态
配置地址
[root@localhost ~]# ifconfig ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 202.202.2.3 netmask 255.255.255.0 broadcast 192.168.3.255 inet6 fe80::20c:29ff:fec4:811 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:c4:08:11 txqueuelen 1000 (Ethernet) RX packets 1745 bytes 254881 (248.9 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1058 bytes 135790 (132.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ens36: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.3.6 netmask 255.255.255.0 broadcast 192.168.3.255 inet6 fe80::20c:29ff:fec4:811 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:c4:08:11 txqueuelen 1000 (Ethernet) RX packets 1745 bytes 254881 (248.9 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1058 bytes 135790 (132.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
安装mariadb数据库
[root@localhost ~]# yum -y install mariadb mariadb-server [root@localhost ~]# systemctl start mariadb mysq[root@localhost ~]# mysqladmin -u root password "123456"
安装zabbix
[root@localhost ~]# yum install -y zabbix-server-mysql zabbix-web-mysql zabbix-agent
安装zabbix之后需要配置数据库，并赋予权限，增加数据库的安全性
[root@localhost ~]# mysql -u root -p Enter password: MariaDB [(none)]> create database zabbix character set utf8 collate utf8_bin; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> grant all privileges on zabbix.* to zabbix@localhost identified by '123456'; Query OK, 0 rows affected (0.00 sec)
导入数据库SQL脚本
1. [root@localhost ~]# zcat /usr/share/doc/zabbix-server-mysql-3.4.1/create.sql.gz | mysql -uzabbix -p zabbix 2. Enter password:
检查编辑配置文件，指定一下用户密码
1. [root@localhost ~]# vim /etc/zabbix/zabbix_server.conf 2. DBPassword=123456
开启zabbix服务
[root@localhost ~]# systemctl start zabbix-server.service [root@localhost ~]# systemctl start zabbix-agent.service #开启客户端 [root@localhost ~]# systemctl enable zabbix-server.service Created symlink from /etc/systemd/system/multi-user.target.wants/zabbix-server.service to /usr/lib/systemd/system/zabbix-server.service.
配置zabbix web接口
[root@localhost ~]# vim /etc/httpd/conf.d/zabbix.conf php_value date.timezone Asia/Shangshai [root@localhost ~]# systemctl start httpd
网页访问192.168.2.6/zabbix/setup.php 安装zabbix服务
八，为lvs和Web服务器安装zabbix客户端
Zabbix添加被监控设备
[root@localhost ~]# yum -y install zabbix-agent
修改agent的配置文件/etc/zabbix/zabbix_agentd.conf。指定Server与ServerActive配置项指定zabbix服务器地址。
[root@localhost ~]# vim /etc/zabbix/zabbix_agentd.conf Server = 192.168.2.1 ServerActive = 192.168.2.1 Hostname = LVS-01
开启agent服务器，默认端口为10050，如果开启防火墙，需要开放响应端口
[root@localhost ~]# systemctl start zabbix-agent.service [root@localhost ~]# netstat -anpt | grep "agent" tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN 14766/zabbix_agentd tcp6 0 0 :::10050 :::* LISTEN 14766/zabbix_agentd
在zabbix服务器上添加主机，配置--》主机--》创建主机--》添加主机名为”LVS-01“，群组为”Linux LVS“的主机。
在zabbix服务器上添加主机，配置--》主机--》创建主机--》添加主机名为”Web-01“，群组为”Linux Web“的主机。
九，zabbix服务器远程控制LVS群集和web群集
在咋zabbix服务器上创建密钥对
[root@localhost ~]# ssh-keygen -t rsa #生成密钥对 Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): #密钥对存放路径 Created directory '/root/.ssh'. #输入私钥保护密码，直接Enter键表示无密码 Enter passphrase (empty for no passphrase): Enter same passphrase again: #再次输入 Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: bb:9a:1a:a0:f4:46:e8:cd:57:94:61:27:1a:79:19:7d root@localhost.localdomain The key's randomart image is: +--[ RSA 2048]----+ | ..=+. | | .+o=. E | | ..o . | | . . | | o.. .S | |o.=. . . | |.. =.. . | | . .. . . | | ..o.. | +-----------------+
将密钥对上传到LVS和Web群集中，实现免密钥登录
[root@localhost ~]# ssh-copy-id root@202.202.2.1 #将密钥对复制到192.168.2.2服务器 The authenticity of host '192.168.2.2 (192.168.2.2)' can't be established. ECDSA key fingerprint is 78:fe:b4:ad:7d:20:29:d4:e4:33:f8:f8:9e:a1:37:c7. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@192.168.2.2's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@192.168.2.2'" and check to make sure that only the key(s) you wanted were added. [root@localhost ~]# ssh-copy-id root@202.202.2.2 #将密钥对复制到192.168.2.3服务器马上 The authenticity of host '192.168.2.3 (192.168.2.3)' can't be established. ECDSA key fingerprint is 92:38:19:c6:28:50:1b:f5:60:5f:04:54:8d:2c:27:81. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@192.168.2.3's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@192.168.2.3'" and check to make sure that only the key(s) you wanted were added. [root@localhost ~]# ssh 202.202.2.1 #实现免密码登录202.202.2.1 Last login: Fri Sep 9 13:36:18 2022 from 192.168.2.88 [root@localhost ~]# exit #退出202.202.2.1服务器 登出 Connection to 202.202.2.6 closed. [root@localhost ~]# ssh 202.202.2.2 #实现免密码登录202.202.2.2 Last login: Fri Sep 9 13:36:43 2022 from 192.168.2.88 [root@localhost ~]# exit #退出202.202.2.2服务器 登出 Connection to 202.202.2.6 closed
所有配置到此完成！！！
