Centos下部署DRBD+NFS+Keepalived高可用环境记录-阿里云开发者社区

开发者社区> 吞吞吐吐的> 正文

Centos下部署DRBD+NFS+Keepalived高可用环境记录

简介:
+关注继续查看

使用NFS服务器(比如图片业务),一台为主,一台为备。通常主到备的数据同步是通过rsync来做(可以结合inotify做实时同步)。由于NFS服务是存在单点的,出于对业务在线率和数据安全的保障,可以采用"DRBD+NFS+Keepalived"架构来完成高可用方案部署。之前介绍了DRBD详细解说及配置过程记录,废话不多说了,基于之前的那篇文档的机器配置信息,以下记录部署过程:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
思路:
1)在两台机器上安装keepalived,VIP为192.168.1.200
2)将DRBD的挂载目录/data作为NFS的挂载目录。远程客户机使用vip地址挂载NFS
3)当Primary主机发生宕机或NFS挂了的故障时,Secondary主机提权升级为DRBD的主节点,并且VIP资源也会转移过来。
   当Primary主机的故障恢复时,会再次变为DRBD的主节点,并重新夺回VIP资源。从而实现故障转移
-----------------------------------------------------------------------------------------------------------
Primary和Secondary两台主机的DRBD环境部署,参见http://www.cnblogs.com/kevingrace/p/5740940.html
  
Primary主机(192.168.1.151)默认作为DRBD的主节点,DRBD挂载目录是/data
Secondary主机(192.168.1.152)是DRBD的备份节点
  
在Primary主机上查看DRBD状态,如下,可知Primary主机是DRBD的主节点
[root@Primary ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res  cs         ro                 ds                 p  mounted  fstype
0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C  /data    ext4
  
如下,DRBD已完成挂载,挂载目录是/data
[root@Primary ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      156G   36G  112G  25% /
tmpfs                 2.9G     0  2.9G   0% /dev/shm
/dev/vda1             190M   98M   83M  55% /boot
/dev/drbd0            9.8G   23M  9.2G   1% /data
  
DRBD数据如下
[root@Primary ~]# cd /data
[root@Primary data]# ll
total 16
-rw-r--r--. 1 root root 9 May 25 09:33 test3
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo1
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo2
  
-----------------------------------------------------------------------------------------------------------
在Primary和Secondary两台主机上安装NFS(可以参考:http://www.cnblogs.com/kevingrace/p/6084604.html)
[root@Primary ~]# yum install rpcbind nfs-utils
[root@Primary ~]# vim /etc/exports
/data 192.168.1.0/24(rw,sync,no_root_squash)
  
[root@Primary ~]# /etc/init.d/rpcbind start
[root@Primary ~]# /etc/init.d/nfs start
---------------------------------------------------------------------------------------------------------
关闭两台主机的iptables防火墙
防火墙最好关闭,否则可能导致客户机挂载nfs时会失败!
若开启防火墙,需要在iptables中开放nfs相关端口机以及VRRP组播地址
[root@Primary ~]# /etc/init.d/iptables stop
 
两台机器上的selinux一定要关闭!!!!!!!!!!
否则下面在keepalived.conf里配置的notify_master.sh等脚本执行失败!这是曾经踩过的坑!
[root@Primary ~]# setenforce 0     //临时关闭。永久关闭的话,还需要在/etc/sysconfig/selinux 文件里将SELINUX改为disabled
[root@Primary ~]# getenforce
Permissive
-----------------------------------------------------------------------------------------------------------
在两台主机上安装Keepalived,配合keepalived实现自动fail-over
  
安装Keepalived
[root@Primary ~]# yum install -y openssl-devel popt-devel
[root@Primary ~]# cd /usr/local/src/
[root@Primary src]# wget http://www.keepalived.org/software/keepalived-1.3.5.tar.gz
[root@Primary src]# tar -zvxf keepalived-1.3.5.tar.gz
[root@Primary src]# cd keepalived-1.3.5
[root@Primary keepalived-1.3.5]# ./configure --prefix=/usr/local/keepalived
[root@Primary keepalived-1.3.5]# make && make install
        
[root@Primary keepalived-1.3.5]# cp /usr/local/src/keepalived-1.3.5/keepalived/etc/init.d/keepalived /etc/rc.d/init.d/
[root@Primary keepalived-1.3.5]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@Primary keepalived-1.3.5]# mkdir /etc/keepalived/
[root@Primary keepalived-1.3.5]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
[root@Primary keepalived-1.3.5]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
[root@Primary keepalived-1.3.5]# echo "/etc/init.d/keepalived start" >> /etc/rc.local
         
[root@Primary keepalived-1.3.5]# chmod +x /etc/rc.d/init.d/keepalived      #添加执行权限
[root@Primary keepalived-1.3.5]# chkconfig keepalived on                   #设置开机启动
[root@Primary keepalived-1.3.5]# service keepalived start                   #启动
[root@Primary keepalived-1.3.5]# service keepalived stop                    #关闭
[root@Primary keepalived-1.3.5]# service keepalived restart                 #重启
  
  
-----------Primary主机的keepalived.conf配置
[root@Primary ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf-bak
[root@Primary ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
  notification_email {
    root@localhost
    }
     
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id DRBD_HA_MASTER
}
     
vrrp_script chk_nfs {
        script "/etc/keepalived/check_nfs.sh"
        interval 5
    }
    vrrp_instance VI_1 {
        state MASTER
        interface eth0
        virtual_router_id 51
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        track_script {
            chk_nfs
        }
    notify_stop /etc/keepalived/notify_stop.sh       
    notify_master /etc/keepalived/notify_master.sh   
    virtual_ipaddress {
        192.168.1.200
    }
}
  
启动keepalived服务
[root@Primary data]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]
[root@Primary data]# ps -ef|grep keepalived
root     30937     1  0 11:49 ?        00:00:00 keepalived -D
root     30939 30937  0 11:49 ?        00:00:00 keepalived -D
root     30940 30937  0 11:49 ?        00:00:00 keepalived -D
root     31123 10364  0 11:50 pts/1    00:00:00 grep --color keepalived
  
查看VIP
[root@Primary data]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:35:d1:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.151/24 brd 192.168.1.255 scope global eth0
    inet 192.168.1.200/32 scope global eth0
    inet6 fe80::f816:3eff:fe35:d1d6/64 scope link
       valid_lft forever preferred_lft forever
  
-----------Secondary主机的keepalived.conf配置
[root@Secondary ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf-bak
[root@Secondary ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
  notification_email {
    root@localhost
    }
     
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id DRBD_HA_BACKUP
}
     
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    notify_master /etc/keepalived/notify_master.sh               //当此机器为keepalived的master角色时执行这个脚本
  
    notify_backup /etc/keepalived/notify_backup.sh               //当此机器为keepalived的backup角色时执行这个脚本
  
    virtual_ipaddress {
        192.168.1.200
    }
}
  
启动keepalived服务
[root@Secondary ~]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]
[root@Secondary ~]# ps -ef|grep keepalived
root     17128     1  0 11:50 ?        00:00:00 keepalived -D
root     17129 17128  0 11:50 ?        00:00:00 keepalived -D
root     17131 17128  0 11:50 ?        00:00:00 keepalived -D
root     17219 29939  0 11:50 pts/1    00:00:00 grep --color keepalived
  
-------------四个脚本配置---------------
1)此脚本只在Primary机器上配置
[root@Primary ~]# vim /etc/keepalived/check_nfs.sh
#!/bin/sh
  
###检查nfs可用性:进程和是否能够挂载
/sbin/service nfs status &>/dev/null
if [ $? -ne 0 ];then
    ###如果服务状态不正常,先尝试重启服务
    /sbin/service nfs restart
    /sbin/service nfs status &>/dev/null
    if [ $? -ne 0 ];then
        ###若重启nfs服务后,仍不正常
        ###卸载drbd设备
        umount /dev/drbd0
        ###将drbd主降级为备
        drbdadm secondary r0
        #关闭keepalived
        /sbin/service keepalived stop
    fi
fi
  
[root@Primary ~]# chmod 755 /etc/keepalived/check_nfs.sh
  
  
2)此脚本只在Primary机器上配置
[root@Primary ~]# mkdir /etc/keepalived/logs
[root@Primary ~]# vim /etc/keepalived/notify_stop.sh
#!/bin/bash
  
time=`date "+%F  %H:%M:%S"`
echo -e "$time  ------notify_stop------\n" >> /etc/keepalived/logs/notify_stop.log
/sbin/service nfs stop &>> /etc/keepalived/logs/notify_stop.log
/bin/umount /data &>> /etc/keepalived/logs/notify_stop.log
/sbin/drbdadm secondary r0 &>> /etc/keepalived/logs/notify_stop.log
echo -e "\n" >> /etc/keepalived/logs/notify_stop.log
  
[root@Primary ~]# chmod 755 /etc/keepalived/notify_stop.sh
  
3)此脚本在两台机器上都要配置
[root@Primary ~]# vim /etc/keepalived/notify_master.sh
#!/bin/bash
  
time=`date "+%F  %H:%M:%S"`
echo -e "$time    ------notify_master------\n" >> /etc/keepalived/logs/notify_master.log
/sbin/drbdadm primary r0 &>> /etc/keepalived/logs/notify_master.log
/bin/mount /dev/drbd0 /data &>> /etc/keepalived/logs/notify_master.log
/sbin/service nfs restart &>> /etc/keepalived/logs/notify_master.log
echo -e "\n" >> /etc/keepalived/logs/notify_master.log
  
[root@Primary ~]# chmod 755 /etc/keepalived/notify_master.sh
  
4)此脚本只在Secondary机器上配置
[root@Secondary ~]# mkdir /etc/keepalived/logs
[root@Secondary ~]# vim /etc/keepalived/notify_backup.sh
#!/bin/bash
  
time=`date "+%F  %H:%M:%S"`
echo -e "$time    ------notify_backup------\n" >> /etc/keepalived/logs/notify_backup.log
/sbin/service nfs stop &>> /etc/keepalived/logs/notify_backup.log
/bin/umount /dev/drbd0 &>> /etc/keepalived/logs/notify_backup.log
/sbin/drbdadm secondary r0 &>> /etc/keepalived/logs/notify_backup.log
echo -e "\n" >> /etc/keepalived/logs/notify_backup.log
  
[root@Secondary ~]# chmod 755 /etc/keepalived/notify_backup.sh
-----------------------------------------------------------------------------------------------------------
在远程客户机上挂载NFS
客户端只需要安装rpcbind程序,并确认服务正常
[root@huanqiu ~]# yum install rpcbind nfs-utils
[root@huanqiu ~]# /etc/init.d/rpcbind start
  
挂载NFS
[root@huanqiu ~]# mount -t nfs 192.168.1.200:/data /web
  
如下查看,发现已经成功挂载了NFS
[root@huanqiu ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      107G   15G   87G  14% /
tmpfs                 2.9G     0  2.9G   0% /dev/shm
/dev/vda1             190M   67M  113M  38% /boot
192.168.1.200:/data   9.8G   23M  9.2G   1% /web
  
[root@huanqiu ~]# cd /web/
[root@huanqiu web]# ll
total 16
-rw-r--r--. 1 root root 9 May 25 09:33 test3
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo1
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo2
-----------------------------------------------------------------------------------------------------------
接着进行fail-over(故障)自动切换测试:
  
1)
先关闭Primary主机上的keepalived服务。就会发现VIP资源已经转移到Secondary主机上了。
同时,Primary主机的nfs也会主动关闭,同时Secondary会升级为DRBD的主节点
[root@Primary ~]# /etc/init.d/keepalived stop
Stopping keepalived:                                       [  OK  ]
[root@Primary ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:35:d1:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.151/24 brd 192.168.1.255 scope global eth0
    inet6 fe80::f816:3eff:fe35:d1d6/64 scope link
       valid_lft forever preferred_lft forever
  
查看系统日志,也能看到VIP资源转移信息
[root@Primary ~]# tail -1000 /var/log/messages
........
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:51 localhost Keepalived[30937]: Stopping
May 25 11:58:51 localhost Keepalived_vrrp[30940]: VRRP_Instance(VI_1) sent 0 priority
May 25 11:58:51 localhost Keepalived_vrrp[30940]: VRRP_Instance(VI_1) removing protocol VIPs.
  
[root@Primary ~]# ps -ef|grep nfs
root       588 10364  0 12:13 pts/1    00:00:00 grep --color nfs
[root@Primary ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      156G   36G  112G  25% /
tmpfs                 2.9G     0  2.9G   0% /dev/shm
/dev/vda1             190M   98M   83M  55% /boot
[root@Primary ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res  cs         ro                   ds                 p  mounted  fstype
0:r0   Connected  Secondary/Secondary  UpToDate/UpToDate  C
  
登录到Secondary备份机器上,发现VIP资源已经转移过来
[root@Secondary ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:4c:7e:88 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.152/24 brd 192.168.1.255 scope global eth0
    inet 192.168.1.200/32 scope global eth0
    inet6 fe80::f816:3eff:fe4c:7e88/64 scope link
       valid_lft forever preferred_lft forever
  
[root@Secondary ~]# tail -1000 /var/log/messages
........
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:58 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:58 localhost Keepalived_vrrp[17131]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.1.200
  
[root@Secondary ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:4c:7e:88 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.152/24 brd 192.168.1.255 scope global eth0
    inet 192.168.1.200/32 scope global eth0
    inet6 fe80::f816:3eff:fe4c:7e88/64 scope link
       valid_lft forever preferred_lft forever
[root@Secondary ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      156G   13G  135G   9% /
tmpfs                 2.9G     0  2.9G   0% /dev/shm
/dev/vda1             190M   89M   92M  50% /boot
/dev/drbd0            9.8G   23M  9.2G   1% /data
  
当Primary机器的keepalived服务恢复启动后,VIP资源又会强制夺回来(可以查看/var/log/message系统日志)
并且Primary还会再次变为DRBD的主节点
  
2)
关闭Primary主机的nfs服务。根据监控脚本,会主动去启动nfs,只要当启动失败时,才会强制由DRBD的主节点降为备份节点,并关闭keepalived。
从而跟上面流程一样实现故障转移
  
结论:
在上面的主从故障切换过程中,对于客户端来说,挂载NFS不影响使用,只是会有一点的延迟。
这也验证了drbd提供的数据一致性功能(包括文件的打开和修改状态等),在客户端看来,真个切换过程就是"一次nfs重启"(主nfs停,备nfs启)。
***************当你发现自己的才华撑不起野心时,就请安静下来学习吧***************
本文转自散尽浮华博客园博客,原文链接:http://www.cnblogs.com/kevingrace/p/5740953.html,如需转载请自行联系原作者

版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

相关文章
阿里云服务器怎么设置密码?怎么停机?怎么重启服务器?
如果在创建实例时没有设置密码,或者密码丢失,您可以在控制台上重新设置实例的登录密码。本文仅描述如何在 ECS 管理控制台上修改实例登录密码。
9494 0
CentOS6.4 LVS+keepalived高可用负载均衡服务配置
<p style="color:rgb(85,85,85)"><span style="font-size:24px; color:#4f81bd"><strong>CentOS6.4 LVS+keepalived高可用负载均衡服务配置</strong></span></p> <p style="color:rgb(85,85,85)"> </p> <p style="color:rg
1343 0
树莓派4开箱记录&公开课预约入口
树莓派4已经上线发售,小编第一时间为大家同步开箱记录,8月30号会用公开课(线上直播)跟大家分享如何用树莓派和物联网平台做设备上云开发。
1970 0
阿里云服务器如何登录?阿里云服务器的三种登录方法
购买阿里云ECS云服务器后如何登录?场景不同,阿里云优惠总结大概有三种登录方式: 登录到ECS云服务器控制台 在ECS云服务器控制台用户可以更改密码、更换系.
13177 0
记录这一刻:百度搜索结果“文件格式:-HTML版”
刚刚百度搜索任何关键词结果都是“文件格式:-HTML版”,现在问题已经修复了。 有看到这一幕的报个名
457 0
4852
文章
0
问答
文章排行榜
最热
最新
相关电子书
更多
《2021云上架构与运维峰会演讲合集》
立即下载
《零基础CSS入门教程》
立即下载
《零基础HTML入门教程》
立即下载