开发者社区> 余二五> 正文
阿里云
为了无法计算的价值
打开APP
阿里云APP内打开

DRDB实现mariadb高可用

简介:
+关注继续查看

------本文大纲

简介

DRBD资源属性

原理

安装DRDB

DRDB实现高可用

============

一、简介

DRBD是一种块设备,可以被用于高可用(HA)之中。它类似于一个网络RAID-1功能。当你将数据写入本地文件系统时,数据还将会被发送到网络中另一台主机上。以相同的形式记录在一个文件系统中。本地(主节点)与远程主机(备节点)的数据可以保证实时同步。当本地系统出现故障时,远程主机上还会保留有一份相同的数据,可以继续使用。在高可用(HA)中使用DRBD功能,可以代替使用一个共享盘阵。因为数据同时存在于本地主机和远程主机上。切换时,远程主机只要使用它上面的那份备份数据,就可以继续进行服务了。

二、DRBD资源属性

resource name

可以使用除空白字符外的任意ACSII表中的字符;

drbd设备

drbd的设备的访问路径,设备文件/dev/drbd#;

disk

各节点为组成此drbd设备所提供的块设备,通常是一个磁盘分区;

网络属性

节点间为了实现跨主机磁盘镜像而使用的网络配置;

注意:用户空间工具与drdb与内核中使用的模块版本要保持一致,只有在使用drbdadm工具时才会读取配置文件,对多个资源的公共配置,可以提取出来只配置一次,通常保存在Common中,此外还有global配置,这种配置跟资源本身没有关系的;

三、原理

wKiom1NhbnjyRON3AAF1bsI5qmM739.jpg

需要知道,DRBD是工作在系统内核空间,而不是用户空间,它直接复制的是二进制数据,这是它速度快的根本原因。

其次,DRBD至少需要两个节点来工作,一主一次。

DRBD的文件同步过程和普通复制过程的不同:

DRBD在数据进入Buffer Cache时,先经过DRBD这一层,复制一份数据经过TCP/IP协议封装,发送到另一个节点上,另一个节点通过TCP/IP协议来接受复制过来的数据,同步到次节点的DRBD设备上。

  • DRBD底层设备支持

DRBD需要构建在底层设备之上,然后构建出一个块设备出来。对于用户来说,一个DRBD设备,就像是一块物理的磁盘,可以在上面内创建文件系统。DRBD所支持的底层设备有以下这些类:

1、一个磁盘,或者是磁盘的某一个分区;

2、一个soft raid 设备;

3、一个LVM的逻辑卷;

  • DRBD在远程传输上支持三种模式:

协议A:异步复制协议

只要主节点完成本地写操作就认为写操作完成,并且需要复制的数据包会被存放本地TCP发送缓存中,当发生failover故障,数据可能会丢失。

协议B:内存同步(半同步,semi-synchronous)复制协议

当本地磁盘的写己经完成,并且复制数据包己经到达对应从节点,此时主节点才认为磁盘写入己经完成,failover不会导致数据丢失,但当所有节点同时出现故障时,主节点上多数最新写入的数据可能丢失。

协议C:同步复制

只有在本地和远程磁盘都确定写入己完成后,主节点才会认为写入完成。这样可确保发生单点故障时不会导致任何数据丢失,如果发生数据丢失,那也只会在所有节点同时存在错误存储时才会发生这种情况

  • 角色、模式

    • 角色
      在drbd构造的集群中,资源具有角色的概念,分别为primary和secondary。
      所有设为primary的资源将不受限制进行读写操作。可以创建文件系统,可以使用裸设备,甚至直接io。所有设为secondary的设备中不能挂载,不能读写

    • 模式

      drbd也有drbd mode:单主模型(主从),双主模型
      在单主模型下drbd可以使用任意的文件系统
      单在双主模型下只能使用集群文件系统,常用的开源的集群文件系统有:ocfs2和gfs2

四、安装DRDB

前提:

设置主机名(要与uname -n一致,主机名可解析)

时间同步

无密钥登录

本机环境

系统 主机名 ip地址
Centos6.5 x86_64 essun.node1.com 192.168.1.109
Centos6.5 x86_64 essun.node2.com 192.168.1.108

注:

内核模块程序包一定要跟你的系统内核保持一致,uname -r查看你的内核版本,内核模块的版本必须要严格对应,而用户空间的模块就不那么严格要求了,两个节点的时间也需要保持一致性;

第一步、下载与内核版本相应的内核模块与用户空间工具(地址:http://packages.atrpms.net/dist/el6/drbd/)

1
2
3
4
5
[root@essun download]# uname -r
2.6.32-431.el6.x86_64
[root@essun download]# ls
drbd-8.4.3-33.el6.x86_64.rpm  drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm
[root@essun download]# yum isntall drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm drbd-8.4.3-33.el6.x86_64.rpm -y
  • 修改配置文件(/etc/drbd.d/global_common.conf)

  • 1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    [root@essun drbd.d]# vim global_common.conf
    [root@essun drbd.d]# cat global_common.conf
    global {
        usage-count no;
    #  是否参加DRBD使用者统计.默认是yes
        # minor-count dialog-refresh disable-ip-verification
    }
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
    common {
        handlers {
            # These are EXAMPLE handlers only.
            # They may have severe implications,
            # like hard resetting the node under certain circumstances.
            # Be careful when chosing your poison.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
            pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
            pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
            local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
            # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
            # split-brain "/usr/lib/drbd/notify-split-brain.sh root";
            # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
            # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
            # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
        }
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
        startup {
            # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
            #wfc-timeout 120;
                     #(等待另一个节点上线的超时时长)
           # degr-wfc-timeout 240;
                    #(等待超时后做降级处理)
            #outdated-wfc-timeout
                    #(过期的等待超时)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
                    # wait-after-sb
                    #(脑裂之后等待多长时长)
        }
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
        options {
            # cpu-mask on-no-data-accessible
        }
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
        disk {
            # size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
            on-io-error detach;
                    # 同步错误的做法是分离
            # disk-drain md-flushes resync-rate resync-after al-extents
                    # c-plan-ahead c-delay-target c-fill-target c-max-rate
                    # c-min-rate disk-timeout
        }
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
        net {
            protocol C;
     # 使用协议C.表示收到远程主机的写入确认后,则认为写入完成.
            cram-hmac-alg "sha1";
            shared-secret "drdb.essun.drdb.org";
     # 设置主备机之间通信使用的信息算法.
            # protocol timeout max-epoch-size max-buffers unplug-watermark
            # connect-int ping-int sndbuf-size rcvbuf-size ko-count
            # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
            # after-sb-1pri after-sb-2pri always-asbp rr-conflict
            # ping-timeout data-integrity-alg tcp-cork on-congestion
            # congestion-fill congestion-extents csums-alg verify-alg
            # use-rle
        }
        syncer {
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
            rate 1000M;
    }
     # 设置主备节点同步时的网络速率最大值,单位是字节.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
    }

    提供物理块设备(两个节点的分区号最好相同),不需要格式化

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
[root@essun drbd.d]# fdisk -cu /dev/s
scd0      sda1      sg0       shm/      snd/      stderr    stdout
sda       sda2      sg1       snapshot  sr0       stdin     systty
[root@essun drbd.d]# fdisk -cu /dev/sda
Command (m for help): p
Disk /dev/sda: 128.8 GB, 128849018880 bytes
255 heads, 63 sectors/track, 15665 cylinders, total 251658240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006c47d
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      411647      204800   83  Linux
/dev/sda2          411648   126240767    62914560   8e  Linux LVM
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First sector (126240768-251658239, default 126240768):
Using default value 126240768
Last sector, +sectors or +size{K,M,G} (126240768-251658239, default 251658239): +10G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@essun drbd.d]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
[root@essun drbd.d]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3
[root@essun drbd.d]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3
[root@essun drbd.d]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3
  • 定义DRBD磁盘资源

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@essun drbd.d]# vim dbresouce.res
resource dbresource {
        on essun.node1.com { #on说明在哪个节点上,跟uname -n保持一致
                device /dev/drbd0#在磁盘上表现的drbd叫什么名;
                disk   /dev/sda3#所使用的物理磁盘设备是哪个;
                address 192.168.1.109:7789; #在node1这个节点上监听的套接字,默认监听在7789端口上;
                meta-disk internal; #保存drbd源数据信息,表示就放在自己的磁盘区分上,也可以放在外部的磁盘上;
        }
        on essun.node2.com{
                device /dev/drbd0;
                disk /dev/sda3;
                address 192.168.1.108:7789;
                meta-disk internal;
        
}
  • 将这两个文件拷贝到node1中的相同目录下

1
2
3
4
5
[root@essun drbd.d]# scp dbresouce.res global_common.conf essun.node1.com:/etc
/drbd.d/
dbresouce.res                               100%  259     0.3KB/s   00:00
global_common.conf                          100% 2044     2.0KB/s   00:00
[root@essun drbd.d]#
  • 在各自的节点上初始化资源

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
[root@essun drbd.d]# drbdadm create-md dbresource
#初始化资源列表
Writing meta data...
initializing activity log
NOT initializing bitmap
lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory
New drbd meta data block successfully created.
lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory
[root@essun drbd.d]# service drbd start
#启动服务
Starting DRBD resources: [
     create res: dbresource
   prepare disk: dbresource
    adjust disk: dbresource
     adjust net: dbresource
]
.
[root@essun drbd.d]# drbd-overview
#查看drbd的状态
  0:dbresource/0  Connected Secondary/Secondary Inconsistent/Inconsistent C r-----
[root@essun drbd.d]# watch -n 1 'cat /proc/drbd'
#实时查看drbd的状态
Every 1.0s: cat /proc/drbd                                                     Thu May  1 08:48:23 2014
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00
 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10485404

上面表示两个节点的drbd都是从的,这是因为我们并没有对其中任何一个提升主的。

  • 将其中一个节点的DRBD资源提升为主资源

1
2
3
4
5
6
[root@essun drbd.d]# drbdadm primary --force dbresource
#将当前节点的drbd资源提升为主资源
[root@essun drbd.d]# drbd
[root@essun drbd.d]# drbd-overview
  0:dbresource/0  SyncSource Primary/Secondary UpToDate/Inconsistent C r---n-
    [>...................] sync'ed:  6.1% (9624/10236)M
  • 初始化之后(角色己经发生改变了)

1
2
[root@essun drbd.d]# drbd-overview
  0:dbresource/0  Connected Primary/Secondary UpToDate/UpToDate C r-----
  • 格式化DRBD

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
[root@essun drbd.d]# mke2fs -t ext4 /dev/drbd0
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2621351 blocks
131067 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Writing inode tables: done                      
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@essun drbd.d]# mount /dev/drbd0 /mnt
[root@essun drbd.d]# cd mnt
-bashcd: mnt: No such file or directory
[root@essun drbd.d]# cd /mnt
[root@essun mnt]# ls
lost+found
[root@essun mnt]# cp /etc/inittab .
[root@essun mnt]# ls
inittab  lost+found
[root@essun mnt]#
  • 提升另一个节点的DRBD资源

1
2
3
4
5
6
7
8
9
10
11
12
[root@essun /]# umount /mnt
#卸载drbd
[root@essun /]# drbdadm secondary dbresource
# 降级DRBD资源
[root@essun /]# drbd-overview
  0:dbresource/0  Connected Secondary/Secondary UpToDate/UpToDate C r-----
#以下是在另一个节点操作
[root@essun ~]# drbdadm primary dbresource
[root@essun ~]# mount /dev/drbd0 /mnt
[root@essun ~]# cd /mnt
[root@essun mnt]# ls
inittab  lost+found

如果想让DRBD能够自动切换的话,就要借助于messages layer程序了。

首先在对DRBD降级,停止服务,禁止开机自动启动

1
2
3
4
5
6
7
8
9
[root@essun /]# umount /mnt
[root@essun /]# drbdadm secondary dbresource
[root@essun /]# drbd-overview
  0:dbresource/0  Connected Secondary/Secondary UpToDate/UpToDate C r-----
[root@essun /]# service drbd stop
Stopping all DRBD resources: .
[root@essun /]# chkconfig drbd off
[root@essun /]# chkconfig --list drbd
drbd            0:off   1:off   2:off   3:off   4:off   5:off   6:off

第二步、安装mariadb

1
2
3
4
5
6
7
[root@essun download]# tar xf mariadb-10.0.10-linux-x86_64.tar.gz
[root@essun download]# mv mariadb-10.0.10-linux-x86_64 /usr/local
[root@essun download]# cd /usr/local/
[root@essun local]# ln -s mariadb-10.0.10-linux-x86_64 mysql
[root@essun local]# groupadd -r mysql
[root@essun local]# useradd -r -g mysql mysql
[root@essun local]# chown -R mysql.mysql mari/
  • 启动DRBD服务,提升主资源

1
2
3
4
5
6
7
8
9
10
11
12
[root@essun ~]# service drbd start
Starting DRBD resources: [
     create res: dbresource
   prepare disk: dbresource
    adjust disk: dbresource
     adjust net: dbresource
]
.
[root@essun local]# drbd-overview
  0:dbresource/0  Connected Secondary/Secondary UpToDate/UpToDate C r-----
[root@essun local]# drbd-overview
  0:dbresource/0  Connected Primary/Secondary UpToDate/UpToDate C r-----
  • 挂载数据目录

1
2
3
4
5
6
7
8
9
10
11
12
[root@essun local]# mkdir /mariadb
[root@essun local]# mount /dev/drbd0 /mariadb/
[root@essun local]# cd /mariadb/
[root@essun mariadb]# mkdir {data,logs}
[root@essun mariadb]# ls
data   logs  lost+found
[root@essun /]# chown -R mysql.mysql /mariadb/
[root@essun /]# ll /mariadb/
total 24
drwxr-xr-x 2 mysql mysql  4096 May  1 11:09 data
drwxr-xr-x 2 mysql mysql  4096 May  1 11:09 logs
drwx------ 2 mysql mysql 16384 May  1 09:42 lost+found

注:

此目录在另一个节点上也要一样。

  • 修改配置文件与添加启动脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
[root@essun mari]# cp support-files/my-large.cnf /etc/cnf
[root@essun mari]# vim /etc/my.cnf
[root@essun mari]# grep -v "^#" /etc/my.cnf |grep -v "
[client]
port        = 3306
socket      = /tmp/mysql.sock
[mysqld]
port        = 3306
socket      = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 256M
max_allowed_packet = 1M
table_open_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M
thread_concurrency = 8
datadir=/mariadb/data
log-bin=/mariadb/logs/mysql-bin
binlog_format=mixed
server-id   = 1
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[myisamchk]
key_buffer_size = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
[root@essun mari]# cp support-files/mysql.server /etc/rc.d/init.d/mari
[root@essun mari]# chmod +x /etc/rc.d/init.d/mari
[root@essun mari]# chkconfig --add mari
[root@essun mari]# chkconfig mari off
  • 初始化数据库,并登录测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@essun mari]# scripts/mysql_install_db --user=mysql --datadir=/mariadb/data/
Installing MariaDB/MySQL system tables in '/mariadb/data/' ...
140501 11:22:15 [Note] InnoDB: The InnoDB memory heap is disabled
140501 11:22:15 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140501 11:22:15 [Note] InnoDB: Compressed tables use zlib 1.2.3
........省略中........
Please report any problems at http://mariadb.org/jira
The latest information about MariaDB is available at http://mariadb.org/.
You can find additional information about the MySQL part at:
http://dev.mysql.com
Support MariaDB development by buying support/new features from
SkySQL Ab. You can contact us about this at sales@skysql.com.
Alternatively consider joining our community based development effort:
http://mariadb.com/kb/en/contributing-to-the-mariadb-project/
[root@essun local]# service mari start
Starting MySQL........                                     [  OK  ]
[root@essun local]# /usr/local/mysql/bin/mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4
Server version: 10.0.10-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
test               |
+--------------------+
4 rows in set (0.00 sec)
MariaDB [(none)]>
  • 到另一个节点测试一下,先停止服务,卸载、降级

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
[root@essun local]# service mari stop
Shutting down MySQL..                                      [  OK  ]
[root@essun local]# umount /mariadb/
[root@essun local]# drbdadm secondary dbresource
[root@essun local]# drbd-overview
  0:dbresource/0  Connected Secondary/Secondary UpToDate/UpToDate C r-----
[root@essun local]# scp /etc/my.cnf essun.node1.com:/etc/
my.cnf                                                               100% 4936     4.8KB/s   00:00
[root@essun local]# scp -p /etc/rc.d/init.d/mari essun.node1.com:/etc/rc.d/init.d/
mari                                                                 100%   11KB  11.4KB/s   00:00
[root@essun local]#
==============以下是另一个节点的操作
[root@essun ~]# mkdir /mariadb
[root@essun ~]# drbdadm primary dbresource
[root@essun ~]# drbd-overview
  0:dbresource/0  Connected Primary/Secondary UpToDate/UpToDate C r-----
[root@essun ~]# chown -R mysql.mysql /mariadb/
[root@essun ~]# mount /dev/drbd0 /mariadb/
[root@essun ~]# cd /mariadb/
[root@essun mariadb]# ll
total 24
drwxr-xr-x 5 mysq mysql  4096 May  1 11:29 data
drwxr-xr-x 2 mysq mysql  4096 May  1 11:29 logs
drwx------ 2 mysq mysql 16384 May  1 09:42 lost+found
[root@essun /]# service mari start
Starting MySQL.......                                      [  OK  ]
[root@essun /]# /usr/local/mysql/bin/mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4
Server version: 10.0.10-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
test               |
+--------------------+
4 rows in set (0.03 sec)
MariaDB [(none)]>
  • 两个节点都要停止服务

1
2
3
4
5
6
7
8
9
10
11
[root@essun /]# service mari stop
Shutting down MySQL..                                      [  OK  ]
[root@essun /]# chkconfig mari off
[root@essun /]# chkconfig --list mari
mari            0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root@essun /]# service drbd stop
Stopping all DRBD resources:
.
[root@essun /]# chkconfig drbd off
[root@essun /]# chkconfig --list drbd
drbd            0:off   1:off   2:off   3:off   4:off   5:off   6:off

第三步、安装corosync、pacemaker

1
2
[root@essun download]# yum install corosync pacemaker -y
[root@essun download]# yum install crmsh-1.2.6-4.el6.x86_64.rpm pssh-2.3.1-2.el6.x86_64.rpm -y
  • 修改配置文件,生成密钥

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
[root@essun /]# cd /etc/corosync/
[root@essun corosync]# ls
corosync.conf.example       service.d
corosync.conf.example.udpu  uidgid.d
[root@essun corosync]# cp corosync.conf.example corosync.conf
[root@essun corosync]# vim corosync.conf
[root@essun corosync]# cat corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
    version: 2
    secauth: off
    threads: 0
    interface {
        ringnumber: 0
        bindnetaddr: 192.168.1.1
        mcastaddr: 226.94.1.1
        mcastport: 5405
        ttl: 1
    }
}
logging {
    fileline: off
    to_stderr: no
    to_logfile: yes
    to_syslog: no
    logfile: /var/log/cluster/corosync.log
    debug: off
    timestamp: on
    logger_subsys {
        subsys: AMF
        debug: off
    }
}
amf {
    mode: disabled
}
service {
        name:   pacemaker
        ver:    0
}
aisexce {
        user:   root
        group:  root
}
[root@essun corosync]# mv /dev/random /dev/x
[root@essun corosync]# mv /dev/urandom /dev/random
[root@essun corosync]# corosync-keygen
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /etc/corosync/authkey.
[root@essun corosync]# mv /dev/random /dev/urandom
[root@essun corosync]# mv /dev/x /dev/random
  • 将这两个文件复制到另一个节点上一份

1
2
3
4
[root@essun corosync]# scp authkey corosync.conf essun.node1.com:/etc/corosync/
authkey                100%  128     0.1KB/s   00:00
corosync.conf          100%  554     0.5KB/s   00:00
[root@essun corosync]#
  • 启动服务(两个节点都要启动服务)

1
2
3
4
5
6
7
8
9
10
11
12
[root@essun corosync]# service corosync start
Starting Corosync Cluster Engine (corosync):            [  OK  ]
[root@essun corosync]# crm status
Last updated: Thu May  1 10:05:25 2014
Last change: Thu May  1 10:05:21 2014 via crmd on essun.node2.com
Stack: classic openais (with plugin)
Current DC: essun.node2.com - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured, 2 expected votes
0 Resources configured
Online: [ essun.node1.com essun.node2.com ]
[root@essun corosync]#
  • 禁用stonith裁决,忽略投票规则

1
2
3
4
5
6
7
8
9
10
11
12
13
crm(live)configure# property stonith-enabled=false
crm(live)configure# property no-quorum-policy=ignore
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node essun.node1.com
node essun.node2.com
property $id="cib-bootstrap-options" \
    dc-version="1.1.10-14.el6_5.3-368c726" \
    cluster-infrastructure="classic openais (with plugin)" \
    expected-quorum-votes="2" \
    stonith-enabled="false" \
    no-quorum-policy="ignore"
  • 定义VIP,mariadb服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
crm(live)configure# primitive vip ocf:heartbeat:IPaddr params ip=192.168.1.150 op monitor interval=20s timeout=20s on-fail=restart
crm(live)configure# verify
crm(live)configure# primitive myserver lsb:mari op monitor interval=20s timeout=20s on-fail=restart
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
crm(live)configure# show
node essun.node1.com \
    attributes standby="off"
node essun.node2.com
primitive myserver lsb:mari \
    op monitor interval="20s" timeout="20s" on-fail="restart"
primitive vip ocf:heartbeat:IPaddr \
    params ip="192.168.1.150" \
    op monitor interval="20s" timeout="20s" on-fail="restart"
property $id="cib-bootstrap-options" \
    dc-version="1.1.10-14.el6_5.3-368c726" \
    cluster-infrastructure="classic openais (with plugin)" \
    expected-quorum-votes="2" \
    stonith-enabled="false" \
    no-quorum-policy="ignore"
  • 配置drdb集群资源

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
crm(live)configure# primitive dbdrbd ocf:linbit:drbd params drbd_resource=dbresource op monitor role=Master interval=50s timeout=30s op monitor role=Slave interval=60s timeout=30s op start timeout=240s interval=0 op stop timeout=100s interval=0
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node essun.node1.com \
        attributes standby="off"
node essun.node2.com
node essun.node1.com \
        attributes standby="off"
node essun.node2.com
primitive dbdrbd ocf:linbit:drbd \
        params drbd_resource="dbresource" \
        op monitor role="Master" interval="50s" timeout="30s" \
        op monitor role="Slave" interval="60s" timeout="30s" \
        op start timeout="240s" interval="0" \
        op stop timeout="100s" interval="0"
primitive myserver lsb:mari \
        op monitor interval="20s" timeout="20s" on-fail="restart"
primitive vip ocf:heartbeat:IPaddr \
        params ip="192.168.1.150" \
        op monitor interval="20s" timeout="20s" on-fail="restart"
property $id="cib-bootstrap-options" \
        dc-version="1.1.10-14.el6_5.3-368c726" \
        cluster-infrastructure="classic openais (with plugin)" \
        expected-quorum-votes="2" \
        stonith-enabled="false" \
        no-quorum-policy="ignore"

drbd需要同时运行在两个节点上,但只能有一个节点(primary/secondary模型)是Master,而另一个节点为Slave;因此,它是一种比较特殊的集群资源,其资源类型为多态(Multi-state)clone类型,即主机节点有Master和Slave之分,且要求服务刚启动时两个节点都处于slave状态,一般来说,只要是定义克隆资源,首先其本身必须是主资源

1
2
3
4
5
6
crm(live)configure# ms ms_dbdrdb dbdrbd meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show ms_dbdrdb
ms ms_dbdrdb dbdrbd \
    meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"

  • 到node1验证一下,看看是不是变成了主资源

1
2
3
4
5
6
7
8
9
10
11
12
[root@essun /]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:bf:c2:82 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.109/24 brd 255.255.255.255 scope global eth0
    inet 192.168.1.150/24 brd 255.255.255.255 scope global secondary eth0
    inet6 fe80::20c:29ff:febf:c282/64 scope link tentative dadfailed
       valid_lft forever preferred_lft forever
  • 定义文件系统资源(以便挂载DRBD的主资源,此处挂载的是mariadb的数据目录)

1
2
3
4
5
6
7
8
9
10
11
crm(live)configure# primitive dbstore ocf:heartbeat:Filesystem params device=/dev/drbd0 directory=/mariadb fstype=ext4 op monitor interval40s timeout=40s op start timeout=60s interval=0 op stop timeout=60 interval=0
crm(live)configure# verify
crm(live)configure# commit
WARNING: CIB changed in the meantime: won't touch it!
Do you still want to commit? y
crm(live)configure# show dbstore
primitive dbstore ocf:heartbeat:Filesystem \
    params device="/dev/drbd0" directory="/mariadb" fstype="ext4" \
    op monitor timeout="40s" interval="0" interval40s \
    op start timeout="60s" interval="0" \
    op stop timeout="60" interval="0
  • 结合drbd的特性,只能在drbd服务将某节点设置为Primary以后方可启动。因此,还需要为这两个资源建立排列约束和顺序约束。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
colocation vip_with_myserver inf: vip myserver
crm(live)configure# colocation myserver_with_dbststore inf: myserver
dbstore
crm(live)configure# colocation dbstore_with_ms_dbdrbd inf: dbstore ms_dbdrdb:Master
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# order ms_dbdrbd_before_dbstore inf: ms_dbdrdb:promote dbstore:start
crm(live)configure# verify
crm(live)configure# order vip_before_myserver inf: vip myserver
crm(live)configure# verify
crm(live)configure# order dbstore_before_myserver inf: dbstore:start myserver:start
crm(live)configure# verify
crm(live)configure# commit
crm(live)# status
Last updated: Thu May  1 15:29:04 2014
Last change: Thu May  1 15:28:55 2014 via crmd on essun.node2.com
Stack: classic openais (with plugin)
Current DC: essun.node2.com - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured, 2 expected votes
5 Resources configured
Online: [ essun.node1.com essun.node2.com ]
 vip    (ocf::heartbeat:IPaddr):        Started essun.node1.com
 myserver       (lsb:mari):     Started essun.node1.com
 Master/Slave Set: ms_dbdrdb [dbdrbd]
     Masters: [ essun.node1.com ]
     Slaves: [ essun.node2.com ]
 dbstore        (ocf::heartbeat:Filesystem):    Started essun.node1.com
  • 到节点node1上看一下maridb下是否己经挂载DRBD的主资源

1
2
3
4
[root@essun mariadb]# hostname
essun.node1.com
[root@essun mariadb]# ls
data  logs  lost+found

这己表示在DRBD己经挂载到node1的mariadb目下

第四步、测试高可用mariadb

  • 授权测试用户

1
2
3
4
5
6
7
8
[root@essun /]# /usr/local/mysql/bin/mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4
Server version: 10.0.10-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> grant all on *.* to root@'192.168.1.%' identified by 'mari';
Query OK, 0 rows affected (0.00 sec)
  • 使用vip登录测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[root@essun mariadb]# hostname
essun.node2.com
[root@essun mariadb]# /usr/local/mysql/bin/mysql -uroot -p -h192.168.1.150
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 6
Server version: 10.0.10-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
test               |
+--------------------+
4 rows in set (0.08 sec)
  • 模拟节点node1故障

  • 1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    crm(live)node# standby essun.node1.com
    [root@essun ~]# crm status
    Last updated: Thu May  1 16:28:59 2014
    Last change: Thu May  1 16:27:34 2014 via crmd on essun.node2.com
    Stack: classic openais (with plugin)
    Current DC: essun.node1.com - partition with quorum
    Version: 1.1.10-14.el6_5.3-368c726
    2 Nodes configured, 2 expected votes
    5 Resources configured
    Node essun.node1.com: standby
    Online: [ essun.node2.com ]
     vip    (ocf::heartbeat:IPaddr):    Started essun.node2.com
     myserver   (lsb:mari): Started essun.node2.com
     Master/Slave Set: ms_dbdrdb [dbdrbd]
         Masters: [ essun.node2.com ]
         Stopped: [ essun.node1.com ]
     dbstore    (ocf::heartbeat:Filesystem):    Started essun.node2.com

    再次使用vip登录测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[root@essun mariadb]# hostname
essun.node1.com
[root@essun mariadb]# /usr/local/mysql/bin/mysql -uroot -p -h192.168.1.150
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 10.0.10-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
test               |
+--------------------+
4 rows in set (1.28 sec)










本文转自 jinlinger 51CTO博客,原文链接:http://blog.51cto.com/essun/1405326,如需转载请自行联系原作者

版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

相关文章
+关注
文章
问答
文章排行榜
最热
最新
相关电子书
更多
Patroni安装部署指南
立即下载
Patroni介绍
立即下载
低代码开发师(初级)实战教程
立即下载