使用NFS服务器(比如图片业务),一台为主,一台为备。通常主到备的数据同步是通过rsync来做(可以结合inotify做实时同步)。由于NFS服务是存在单点的,出于对业务在线率和数据安全的保障,可以采用"DRBD+NFS+Keepalived"架构来完成高可用方案部署。之前介绍了DRBD详细解说及配置过程记录,废话不多说了,基于之前的那篇文档的机器配置信息,以下记录部署过程:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
|
思路:
1)在两台机器上安装keepalived,VIP为192.168.1.200
2)将DRBD的挂载目录
/data
作为NFS的挂载目录。远程客户机使用vip地址挂载NFS
3)当Primary主机发生宕机或NFS挂了的故障时,Secondary主机提权升级为DRBD的主节点,并且VIP资源也会转移过来。
当Primary主机的故障恢复时,会再次变为DRBD的主节点,并重新夺回VIP资源。从而实现故障转移
-----------------------------------------------------------------------------------------------------------
Primary和Secondary两台主机的DRBD环境部署,参见http:
//www
.cnblogs.com
/kevingrace/p/5740940
.html
Primary主机(192.168.1.151)默认作为DRBD的主节点,DRBD挂载目录是
/data
Secondary主机(192.168.1.152)是DRBD的备份节点
在Primary主机上查看DRBD状态,如下,可知Primary主机是DRBD的主节点
[root@Primary ~]
# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88
/proto
:86-97)
GIT-
hash
: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res cs ro ds p mounted fstype
0:r0 Connected Primary
/Secondary
UpToDate
/UpToDate
C
/data
ext4
如下,DRBD已完成挂载,挂载目录是
/data
[root@Primary ~]
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
156G 36G 112G 25% /
tmpfs 2.9G 0 2.9G 0%
/dev/shm
/dev/vda1
190M 98M 83M 55%
/boot
/dev/drbd0
9.8G 23M 9.2G 1%
/data
DRBD数据如下
[root@Primary ~]
# cd /data
[root@Primary data]
# ll
total 16
-rw-r--r--. 1 root root 9 May 25 09:33 test3
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo1
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo2
-----------------------------------------------------------------------------------------------------------
在Primary和Secondary两台主机上安装NFS(可以参考:http:
//www
.cnblogs.com
/kevingrace/p/6084604
.html)
[root@Primary ~]
# yum install rpcbind nfs-utils
[root@Primary ~]
# vim /etc/exports
/data
192.168.1.0
/24
(rw,
sync
,no_root_squash)
[root@Primary ~]
# /etc/init.d/rpcbind start
[root@Primary ~]
# /etc/init.d/nfs start
---------------------------------------------------------------------------------------------------------
关闭两台主机的iptables防火墙
防火墙最好关闭,否则可能导致客户机挂载nfs时会失败!
若开启防火墙,需要在iptables中开放nfs相关端口机以及VRRP组播地址
[root@Primary ~]
# /etc/init.d/iptables stop
两台机器上的selinux一定要关闭!!!!!!!!!!
否则下面在keepalived.conf里配置的notify_master.sh等脚本执行失败!这是曾经踩过的坑!
[root@Primary ~]
# setenforce 0 //临时关闭。永久关闭的话,还需要在/etc/sysconfig/selinux 文件里将SELINUX改为disabled
[root@Primary ~]
# getenforce
Permissive
-----------------------------------------------------------------------------------------------------------
在两台主机上安装Keepalived,配合keepalived实现自动fail-over
安装Keepalived
[root@Primary ~]
# yum install -y openssl-devel popt-devel
[root@Primary ~]
# cd /usr/local/src/
[root@Primary src]
# wget http://www.keepalived.org/software/keepalived-1.3.5.tar.gz
[root@Primary src]
# tar -zvxf keepalived-1.3.5.tar.gz
[root@Primary src]
# cd keepalived-1.3.5
[root@Primary keepalived-1.3.5]
# ./configure --prefix=/usr/local/keepalived
[root@Primary keepalived-1.3.5]
# make && make install
[root@Primary keepalived-1.3.5]
# cp /usr/local/src/keepalived-1.3.5/keepalived/etc/init.d/keepalived /etc/rc.d/init.d/
[root@Primary keepalived-1.3.5]
# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@Primary keepalived-1.3.5]
# mkdir /etc/keepalived/
[root@Primary keepalived-1.3.5]
# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
[root@Primary keepalived-1.3.5]
# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
[root@Primary keepalived-1.3.5]
# echo "/etc/init.d/keepalived start" >> /etc/rc.local
[root@Primary keepalived-1.3.5]
# chmod +x /etc/rc.d/init.d/keepalived #添加执行权限
[root@Primary keepalived-1.3.5]
# chkconfig keepalived on #设置开机启动
[root@Primary keepalived-1.3.5]
# service keepalived start #启动
[root@Primary keepalived-1.3.5]
# service keepalived stop #关闭
[root@Primary keepalived-1.3.5]
# service keepalived restart #重启
-----------Primary主机的keepalived.conf配置
[root@Primary ~]
# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf-bak
[root@Primary ~]
# vim /etc/keepalived/keepalived.conf
! Configuration File
for
keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id DRBD_HA_MASTER
}
vrrp_script chk_nfs {
script
"/etc/keepalived/check_nfs.sh"
interval 5
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_nfs
}
notify_stop
/etc/keepalived/notify_stop
.sh
notify_master
/etc/keepalived/notify_master
.sh
virtual_ipaddress {
192.168.1.200
}
}
启动keepalived服务
[root@Primary data]
# /etc/init.d/keepalived start
Starting keepalived: [ OK ]
[root@Primary data]
# ps -ef|grep keepalived
root 30937 1 0 11:49 ? 00:00:00 keepalived -D
root 30939 30937 0 11:49 ? 00:00:00 keepalived -D
root 30940 30937 0 11:49 ? 00:00:00 keepalived -D
root 31123 10364 0 11:50 pts
/1
00:00:00
grep
--color keepalived
查看VIP
[root@Primary data]
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link
/loopback
00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1
/8
scope host lo
inet6 ::1
/128
scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link
/ether
fa:16:3e:35:d1:d6 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.151
/24
brd 192.168.1.255 scope global eth0
inet 192.168.1.200
/32
scope global eth0
inet6 fe80::f816:3eff:fe35:d1d6
/64
scope link
valid_lft forever preferred_lft forever
-----------Secondary主机的keepalived.conf配置
[root@Secondary ~]
# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf-bak
[root@Secondary ~]
# vim /etc/keepalived/keepalived.conf
! Configuration File
for
keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id DRBD_HA_BACKUP
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
notify_master
/etc/keepalived/notify_master
.sh
//
当此机器为keepalived的master角色时执行这个脚本
notify_backup
/etc/keepalived/notify_backup
.sh
//
当此机器为keepalived的backup角色时执行这个脚本
virtual_ipaddress {
192.168.1.200
}
}
启动keepalived服务
[root@Secondary ~]
# /etc/init.d/keepalived start
Starting keepalived: [ OK ]
[root@Secondary ~]
# ps -ef|grep keepalived
root 17128 1 0 11:50 ? 00:00:00 keepalived -D
root 17129 17128 0 11:50 ? 00:00:00 keepalived -D
root 17131 17128 0 11:50 ? 00:00:00 keepalived -D
root 17219 29939 0 11:50 pts
/1
00:00:00
grep
--color keepalived
-------------四个脚本配置---------------
1)此脚本只在Primary机器上配置
[root@Primary ~]
# vim /etc/keepalived/check_nfs.sh
#!/bin/sh
###检查nfs可用性:进程和是否能够挂载
/sbin/service
nfs status &>
/dev/null
if
[ $? -
ne
0 ];
then
###如果服务状态不正常,先尝试重启服务
/sbin/service
nfs restart
/sbin/service
nfs status &>
/dev/null
if
[ $? -
ne
0 ];
then
###若重启nfs服务后,仍不正常
###卸载drbd设备
umount
/dev/drbd0
###将drbd主降级为备
drbdadm secondary r0
#关闭keepalived
/sbin/service
keepalived stop
fi
fi
[root@Primary ~]
# chmod 755 /etc/keepalived/check_nfs.sh
2)此脚本只在Primary机器上配置
[root@Primary ~]
# mkdir /etc/keepalived/logs
[root@Primary ~]
# vim /etc/keepalived/notify_stop.sh
#!/bin/bash
time
=`
date
"+%F %H:%M:%S"
`
echo
-e
"$time ------notify_stop------\n"
>>
/etc/keepalived/logs/notify_stop
.log
/sbin/service
nfs stop &>>
/etc/keepalived/logs/notify_stop
.log
/bin/umount
/data
&>>
/etc/keepalived/logs/notify_stop
.log
/sbin/drbdadm
secondary r0 &>>
/etc/keepalived/logs/notify_stop
.log
echo
-e
"\n"
>>
/etc/keepalived/logs/notify_stop
.log
[root@Primary ~]
# chmod 755 /etc/keepalived/notify_stop.sh
3)此脚本在两台机器上都要配置
[root@Primary ~]
# vim /etc/keepalived/notify_master.sh
#!/bin/bash
time
=`
date
"+%F %H:%M:%S"
`
echo
-e
"$time ------notify_master------\n"
>>
/etc/keepalived/logs/notify_master
.log
/sbin/drbdadm
primary r0 &>>
/etc/keepalived/logs/notify_master
.log
/bin/mount
/dev/drbd0
/data
&>>
/etc/keepalived/logs/notify_master
.log
/sbin/service
nfs restart &>>
/etc/keepalived/logs/notify_master
.log
echo
-e
"\n"
>>
/etc/keepalived/logs/notify_master
.log
[root@Primary ~]
# chmod 755 /etc/keepalived/notify_master.sh
4)此脚本只在Secondary机器上配置
[root@Secondary ~]
# mkdir /etc/keepalived/logs
[root@Secondary ~]
# vim /etc/keepalived/notify_backup.sh
#!/bin/bash
time
=`
date
"+%F %H:%M:%S"
`
echo
-e
"$time ------notify_backup------\n"
>>
/etc/keepalived/logs/notify_backup
.log
/sbin/service
nfs stop &>>
/etc/keepalived/logs/notify_backup
.log
/bin/umount
/dev/drbd0
&>>
/etc/keepalived/logs/notify_backup
.log
/sbin/drbdadm
secondary r0 &>>
/etc/keepalived/logs/notify_backup
.log
echo
-e
"\n"
>>
/etc/keepalived/logs/notify_backup
.log
[root@Secondary ~]
# chmod 755 /etc/keepalived/notify_backup.sh
-----------------------------------------------------------------------------------------------------------
在远程客户机上挂载NFS
客户端只需要安装rpcbind程序,并确认服务正常
[root@huanqiu ~]
# yum install rpcbind nfs-utils
[root@huanqiu ~]
# /etc/init.d/rpcbind start
挂载NFS
[root@huanqiu ~]
# mount -t nfs 192.168.1.200:/data /web
如下查看,发现已经成功挂载了NFS
[root@huanqiu ~]
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
107G 15G 87G 14% /
tmpfs 2.9G 0 2.9G 0%
/dev/shm
/dev/vda1
190M 67M 113M 38%
/boot
192.168.1.200:
/data
9.8G 23M 9.2G 1%
/web
[root@huanqiu ~]
# cd /web/
[root@huanqiu web]
# ll
total 16
-rw-r--r--. 1 root root 9 May 25 09:33 test3
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo1
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo2
-----------------------------------------------------------------------------------------------------------
接着进行fail-over(故障)自动切换测试:
1)
先关闭Primary主机上的keepalived服务。就会发现VIP资源已经转移到Secondary主机上了。
同时,Primary主机的nfs也会主动关闭,同时Secondary会升级为DRBD的主节点
[root@Primary ~]
# /etc/init.d/keepalived stop
Stopping keepalived: [ OK ]
[root@Primary ~]
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link
/loopback
00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1
/8
scope host lo
inet6 ::1
/128
scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link
/ether
fa:16:3e:35:d1:d6 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.151
/24
brd 192.168.1.255 scope global eth0
inet6 fe80::f816:3eff:fe35:d1d6
/64
scope link
valid_lft forever preferred_lft forever
查看系统日志,也能看到VIP资源转移信息
[root@Primary ~]
# tail -1000 /var/log/messages
........
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0
for
192.168.1.200
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0
for
192.168.1.200
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0
for
192.168.1.200
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0
for
192.168.1.200
May 25 11:58:51 localhost Keepalived[30937]: Stopping
May 25 11:58:51 localhost Keepalived_vrrp[30940]: VRRP_Instance(VI_1) sent 0 priority
May 25 11:58:51 localhost Keepalived_vrrp[30940]: VRRP_Instance(VI_1) removing protocol VIPs.
[root@Primary ~]
# ps -ef|grep nfs
root 588 10364 0 12:13 pts
/1
00:00:00
grep
--color nfs
[root@Primary ~]
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
156G 36G 112G 25% /
tmpfs 2.9G 0 2.9G 0%
/dev/shm
/dev/vda1
190M 98M 83M 55%
/boot
[root@Primary ~]
# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88
/proto
:86-97)
GIT-
hash
: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res cs ro ds p mounted fstype
0:r0 Connected Secondary
/Secondary
UpToDate
/UpToDate
C
登录到Secondary备份机器上,发现VIP资源已经转移过来
[root@Secondary ~]
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link
/loopback
00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1
/8
scope host lo
inet6 ::1
/128
scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link
/ether
fa:16:3e:4c:7e:88 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.152
/24
brd 192.168.1.255 scope global eth0
inet 192.168.1.200
/32
scope global eth0
inet6 fe80::f816:3eff:fe4c:7e88
/64
scope link
valid_lft forever preferred_lft forever
[root@Secondary ~]
# tail -1000 /var/log/messages
........
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0
for
192.168.1.200
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0
for
192.168.1.200
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0
for
192.168.1.200
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0
for
192.168.1.200
May 25 11:58:58 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0
for
192.168.1.200
May 25 11:58:58 localhost Keepalived_vrrp[17131]: VRRP_Instance(VI_1) Sending
/queueing
gratuitous ARPs on eth0
for
192.168.1.200
[root@Secondary ~]
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link
/loopback
00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1
/8
scope host lo
inet6 ::1
/128
scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link
/ether
fa:16:3e:4c:7e:88 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.152
/24
brd 192.168.1.255 scope global eth0
inet 192.168.1.200
/32
scope global eth0
inet6 fe80::f816:3eff:fe4c:7e88
/64
scope link
valid_lft forever preferred_lft forever
[root@Secondary ~]
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
156G 13G 135G 9% /
tmpfs 2.9G 0 2.9G 0%
/dev/shm
/dev/vda1
190M 89M 92M 50%
/boot
/dev/drbd0
9.8G 23M 9.2G 1%
/data
当Primary机器的keepalived服务恢复启动后,VIP资源又会强制夺回来(可以查看
/var/log/message
系统日志)
并且Primary还会再次变为DRBD的主节点
2)
关闭Primary主机的nfs服务。根据监控脚本,会主动去启动nfs,只要当启动失败时,才会强制由DRBD的主节点降为备份节点,并关闭keepalived。
从而跟上面流程一样实现故障转移
结论:
在上面的主从故障切换过程中,对于客户端来说,挂载NFS不影响使用,只是会有一点的延迟。
这也验证了drbd提供的数据一致性功能(包括文件的打开和修改状态等),在客户端看来,真个切换过程就是
"一次nfs重启"
(主nfs停,备nfs启)。
|
***************当你发现自己的才华撑不起野心时,就请安静下来学习吧***************
本文转自散尽浮华博客园博客,原文链接:http://www.cnblogs.com/kevingrace/p/5740953.html
,如需转载请自行联系原作者