在rhel5中自带了集群软件,我个人认为不好用,还不如直接编辑配置文件效率高,快!下面是我在rhel5中使用LVS和heartbeat组合来实现集群和负载均衡。
简要说明:服务器四台,server3和server4上两块网卡,分别为:
服务器名称 IP(eth0) IP(eth1)
server3.rhel5.net 192.168.1.30 10.0.0.2
server4.rhel5.net 192.168.1.40 10.0.0.1
server5.rhel5.net 192.168.1.50
server7.rhel5.net 192.168.1.71
虚拟ip为192.168.1.70,server3和server4主要提供集群和负载均衡服务,server3为主,server4为从,server5和server7提供httpd服务。
心跳使用的eth1网卡。
heartbeat软件包下载地址,可以在网上找,一大堆,至于ipvsadm软件包,rhel5光盘中有,自行安装。
所需软件包:heartbeat、libnet、net-snmp、net-snmp-libs、perl-Compress-Zlib、perl-HTML-Parser、perl-HTML-Tagset、perl-libwww-perl、perl-MailTools、perl-TimeDate、perl-URI、ipvsadm.以上软件包,有的在rhel5光盘中,有的要在网上搜索的。至于heartbeat、libnet软件包可以到如下网址下载:
http://www.packetfactory.net/libnet/dist/libnet.tar.gz
建议先安装以上除heartbeat、libnet之外的其他软件包,然后安装heartbeat、libnet.
建议先安装以上除heartbeat、libnet之外的其他软件包,然后安装heartbeat、libnet.
下来直接从安装heartbeat、libnet开始:
在server3上首先安装libnet,否则在安装heartbeat软件时会报错的。
[root@server3 ~]#tar -xvzf libnet.tar.gz
[root@server3 ~]#cd libnet
[root@server3 ~]#./configure
[root@server3 ~]#make;make install
[root@server3 ~]# tar -xvzf heartbeat-2.1.3.tar.gz
[root@server3 ~]# cd heartbeat-2.1.3
[root@server3 ~]# ./ConfigureMe
[root@server3 ~]# make;make install
安装这两个软件包过程比较慢,请耐心等待;安装完了后,在/etc/ha.d目录下默认没有ldirectord.cf文件,我们复制heartbeat-2.1.3/ldirectord/ldirectord.cf文件到/etc/ha.d目录下。
下来编辑ldirectord.cf、authkeys、ha.cf、haresources四个文件。
我的配置如下所示:
ldirectord.cf文件:
checktimeout=3 #检测超时3s
checkinterval=1 #检查时间间隔1s
autoreload=yes #配置文件改变时自动加载配置文件
logfile="/var/log/ldirectord.log" #定义日志文件
quiescent=yes
virtual=192.168.1.70:80 #虚拟IP
real=192.168.1.50:80 gate #realserver server5
real=192.168.1.71:80 gate #realserver server7
fallback=127.0.0.1:80 #所有服务器发生故障时访问该服务器
service=http #服务名称
scheduler=rr #我这里使用的直连路由调度算法
protocol=tcp
checktype=negotiate
checkinterval=1 #检查时间间隔1s
autoreload=yes #配置文件改变时自动加载配置文件
logfile="/var/log/ldirectord.log" #定义日志文件
quiescent=yes
virtual=192.168.1.70:80 #虚拟IP
real=192.168.1.50:80 gate #realserver server5
real=192.168.1.71:80 gate #realserver server7
fallback=127.0.0.1:80 #所有服务器发生故障时访问该服务器
service=http #服务名称
scheduler=rr #我这里使用的直连路由调度算法
protocol=tcp
checktype=negotiate
ha.cf文件:
debugfile /var/log/ha-debug #定义日志文件,其实这个文件的内容与ldirectord.cf文件中定义的/var/log/ldirectord.log文件内容相同。
logfile /var/log/ha-log #同上
logfacility local0 #默认
keepalive 2 #设定heartbeat之间的时间间隔为2秒.
deadtime 30 #在30秒后宣布节点死亡。
warntime 10 #在日志中发出“late heartbeat“警告之前等待的时间,单位为秒。
initdead 120 #在某些配置下,重启后网络需要一些时间才能正常工作。这个单独的”deadtime”选项可以处理这种情况。它的取值至少应该为通常deadtime的两倍。
udpport 694 # 使用端口694进行bcast和ucast通信。这是默认的,并且在IANA官方注册的端口号。
bcast eth1 # Linux 表示在eth1接口上使用广播heartbeat(将eth1替换为eth0,eth2),或者您使用的任何接口。
mcast eth0 225.0.0.1 694 1 0 #默认即可
ucast eth0 192.168.1.30 #改为你eth0的地址。
auto_failback on #该选项是必须配置的。值为on or off
node server3.rhel5.net #负载均衡服务器名,必须与uname -n的输出一致。
node server4.rhel5.net #同上
ping 10.0.0.1
respawn hacluster /usr/lib/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster
logfile /var/log/ha-log #同上
logfacility local0 #默认
keepalive 2 #设定heartbeat之间的时间间隔为2秒.
deadtime 30 #在30秒后宣布节点死亡。
warntime 10 #在日志中发出“late heartbeat“警告之前等待的时间,单位为秒。
initdead 120 #在某些配置下,重启后网络需要一些时间才能正常工作。这个单独的”deadtime”选项可以处理这种情况。它的取值至少应该为通常deadtime的两倍。
udpport 694 # 使用端口694进行bcast和ucast通信。这是默认的,并且在IANA官方注册的端口号。
bcast eth1 # Linux 表示在eth1接口上使用广播heartbeat(将eth1替换为eth0,eth2),或者您使用的任何接口。
mcast eth0 225.0.0.1 694 1 0 #默认即可
ucast eth0 192.168.1.30 #改为你eth0的地址。
auto_failback on #该选项是必须配置的。值为on or off
node server3.rhel5.net #负载均衡服务器名,必须与uname -n的输出一致。
node server4.rhel5.net #同上
ping 10.0.0.1
respawn hacluster /usr/lib/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster
haresources文件:
server3.rhel5.net ldirectord::ldirectord.cf LVSSyncDaemonSwap::master IPaddr::192.168.1.70/24/eth0/192.168.1.255
authkeys文件,这里我使用的是md5方式,注意,该文件的权限必须是600:
auth 3
3 md5 test
3 md5 test
然后编辑/etc/sysctl.conf文件,加入以下行:
net.ipv4.ip_forward = 1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.eth0.arp_announce = 2
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.eth0.arp_announce = 2
再执行sysctl -p命令,使其立即生效!
以上是server3上的配置,下来进入server4上,重复以上安装配置,
切记,server4上的/etc/ha.d/目录下的authkeys、haresources、ha.cf、ldirectord.cf四个文件必须和server3上的相同。
完成后先启动server3上的heartbeat服务,查看日志,是否有报错,然后在启动server4上的heartbeat服务,检查日志文件,是否有报错.
server3上:
[root@server3 ~]ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:8E:52:05
inet addr:192.168.1.30 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe8e:5205/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:113693 errors:0 dropped:0 overruns:0 frame:0
TX packets:137301 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:12951968 (12.3 MiB) TX bytes:11335533 (10.8 MiB)
Interrupt:169 Base address:0x2000
inet addr:192.168.1.30 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe8e:5205/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:113693 errors:0 dropped:0 overruns:0 frame:0
TX packets:137301 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:12951968 (12.3 MiB) TX bytes:11335533 (10.8 MiB)
Interrupt:169 Base address:0x2000
eth0:0 Link encap:Ethernet HWaddr 00:0C:29:8E:52:05
inet addr:192.168.1.70 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:169 Base address:0x2000
inet addr:192.168.1.70 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:169 Base address:0x2000
eth1 Link encap:Ethernet HWaddr 00:0C:29:8E:52:0F
inet addr:10.0.0.2 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::20c:29ff:fe8e:520f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3680 errors:0 dropped:0 overruns:0 frame:0
TX packets:3853 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:720033 (703.1 KiB) TX bytes:748019 (730.4 KiB)
Interrupt:193 Base address:0x2080
inet addr:10.0.0.2 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::20c:29ff:fe8e:520f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3680 errors:0 dropped:0 overruns:0 frame:0
TX packets:3853 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:720033 (703.1 KiB) TX bytes:748019 (730.4 KiB)
Interrupt:193 Base address:0x2080
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:3982 errors:0 dropped:0 overruns:0 frame:0
TX packets:3982 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1010816 (987.1 KiB) TX bytes:1010816 (987.1 KiB)
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:3982 errors:0 dropped:0 overruns:0 frame:0
TX packets:3982 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1010816 (987.1 KiB) TX bytes:1010816 (987.1 KiB)
server4上:
[root@server4 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:FF:1A:AB
inet addr:192.168.1.40 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feff:1aab/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2983 errors:0 dropped:0 overruns:0 frame:0
TX packets:4766 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:617725 (603.2 KiB) TX bytes:1118790 (1.0 MiB)
Interrupt:169 Base address:0x2000
eth0 Link encap:Ethernet HWaddr 00:0C:29:FF:1A:AB
inet addr:192.168.1.40 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feff:1aab/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2983 errors:0 dropped:0 overruns:0 frame:0
TX packets:4766 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:617725 (603.2 KiB) TX bytes:1118790 (1.0 MiB)
Interrupt:169 Base address:0x2000
eth1 Link encap:Ethernet HWaddr 00:0C:29:FF:1A:B5
inet addr:10.0.0.1 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::20c:29ff:feff:1ab5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4135 errors:0 dropped:0 overruns:0 frame:0
TX packets:4070 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:813997 (794.9 KiB) TX bytes:790753 (772.2 KiB)
Interrupt:193 Base address:0x2080
inet addr:10.0.0.1 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::20c:29ff:feff:1ab5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4135 errors:0 dropped:0 overruns:0 frame:0
TX packets:4070 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:813997 (794.9 KiB) TX bytes:790753 (772.2 KiB)
Interrupt:193 Base address:0x2080
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:3374 errors:0 dropped:0 overruns:0 frame:0
TX packets:3374 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:431408 (421.2 KiB) TX bytes:431408 (421.2 KiB)
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:3374 errors:0 dropped:0 overruns:0 frame:0
TX packets:3374 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:431408 (421.2 KiB) TX bytes:431408 (421.2 KiB)
server3上多了个eth0:0接口,若你的没有出现,请等待一会,再使用ifconfig查看,若还是没有eth0:0接口出现,请检查配置文件是否正确!若停止server3上的heartbeat服务,则server4会接管,并产生一个eth0:0接口!
[root@server3 ~]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master running
(ipvs_syncmaster pid: 9132)
[root@server3 ~]# /etc/ha.d/resource.d/LVSSyncDaemonSwap backup status
backup stopped
(ipvs_syncmaster pid: 9132)
master running
(ipvs_syncmaster pid: 9132)
[root@server3 ~]# /etc/ha.d/resource.d/LVSSyncDaemonSwap backup status
backup stopped
(ipvs_syncmaster pid: 9132)
server4上:
[root@server4 ~]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master stopped
(ipvs_syncbackup pid: 5830)
master stopped
(ipvs_syncbackup pid: 5830)
[root@server4 ~]# /etc/ha.d/resource.d/LVSSyncDaemonSwap backup status
backup running
(ipvs_syncbackup pid: 5830)
backup running
(ipvs_syncbackup pid: 5830)
现在停止server3上的heartbeat服务,看server4是否可以正常接管,可以查看日志文件。若没有报错,则说明heartbeat服务没有问题,若存在问题请根据日志文件进行排错!
然后启动server3和server4上的ipvsadm服务,使用ipvsadm命令查看:
[root@server3 ha.d]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.1.70:80 rr
-> 192.168.1.50:80 Route 1 0 0
-> 192.168.1.71:80 Route 1 0 0
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.1.70:80 rr
-> 192.168.1.50:80 Route 1 0 0
-> 192.168.1.71:80 Route 1 0 0
server4上的输出应该和server3上的一致。ok,以上是heartbeat与LVS配置。下面是realserver上的配置:
server5和server7上的配置:
[root@server5 ~]# echo "This is server5">/var/www/html/index.html
[root@server5 ~]# vi /etc/sysctl.conf加入以下行:
net.ipv4.ip_forward = 1
然后编写如下脚本,我这里命名为lvs.sh
#!/bin/sh
echo "1">/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2">/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1">/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2">/proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 192.168.1.70 broadcast 192.168.1.70 netmask 255.255.255.255 up
route add -host 192.168.1.70 dev lo:0
sysctl -p
echo "1">/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2">/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1">/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2">/proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 192.168.1.70 broadcast 192.168.1.70 netmask 255.255.255.255 up
route add -host 192.168.1.70 dev lo:0
sysctl -p
然后在/etc/rc.local文件中把lvs.sh文件的绝对路径,让其在开机是运行。
[root@server7 ~]# echo "This is server7">/var/www/html/index.html
然后把httpd设置为在开机时就启动,chkconfig httpd on.在server7上切记不要忘记那个lvs脚本!重启server5和server7,启动后,检查一下是否多了个网络接口lo:0.我的如下:
lo:0 Link encap:Local Loopback
inet addr:192.168.1.70 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
inet addr:192.168.1.70 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
再查看一下,你的httpd服务是否启动。
在其他机器上的IE里输入
http://192.168.1.70,看是否可以访问,我的成功显示This is server5,然后刷新一下,显示server7。
在server3上使用ipvsadm -Lnc查看那些客户端通过192.168.70访问realserver上的httpd服务:
[root@server3 ~]# ipvsadm -Lnc
IPVS connection entries
pro expire state source virtual destination
TCP 01:52 FIN_WAIT 192.168.1.156:4204 192.168.1.70:80 192.168.1.71:80
TCP 01:53 FIN_WAIT 192.168.1.156:4207 192.168.1.70:80 192.168.1.71:80
TCP 01:51 FIN_WAIT 192.168.1.156:4203 192.168.1.70:80 192.168.1.50:80
TCP 01:53 FIN_WAIT 192.168.1.156:4208 192.168.1.70:80 192.168.1.50:80
TCP 01:50 FIN_WAIT 192.168.1.156:4202 192.168.1.70:80 192.168.1.71:80
TCP 01:53 FIN_WAIT 192.168.1.156:4205 192.168.1.70:80 192.168.1.50:80
IPVS connection entries
pro expire state source virtual destination
TCP 01:52 FIN_WAIT 192.168.1.156:4204 192.168.1.70:80 192.168.1.71:80
TCP 01:53 FIN_WAIT 192.168.1.156:4207 192.168.1.70:80 192.168.1.71:80
TCP 01:51 FIN_WAIT 192.168.1.156:4203 192.168.1.70:80 192.168.1.50:80
TCP 01:53 FIN_WAIT 192.168.1.156:4208 192.168.1.70:80 192.168.1.50:80
TCP 01:50 FIN_WAIT 192.168.1.156:4202 192.168.1.70:80 192.168.1.71:80
TCP 01:53 FIN_WAIT 192.168.1.156:4205 192.168.1.70:80 192.168.1.50:80
ok,以上说明192.168.1.156通过192.168.1.70访问192.168.1.50和192.168.1.71的httpd服务!
Good luck for you!
本文转自legendfu51CTO博客,原文链接:http://blog.51cto.com/legendfu/1072315
,如需转载请自行联系原作者