DRDB实现mariadb高可用-阿里云开发者社区

------本文大纲

简介

DRBD资源属性

原理

安装DRDB

DRDB实现高可用

============

一、简介

DRBD是一种块设备，可以被用于高可用(HA)之中。它类似于一个网络RAID-1功能。当你将数据写入本地文件系统时，数据还将会被发送到网络中另一台主机上。以相同的形式记录在一个文件系统中。本地(主节点)与远程主机(备节点)的数据可以保证实时同步。当本地系统出现故障时，远程主机上还会保留有一份相同的数据，可以继续使用。在高可用(HA)中使用DRBD功能，可以代替使用一个共享盘阵。因为数据同时存在于本地主机和远程主机上。切换时，远程主机只要使用它上面的那份备份数据，就可以继续进行服务了。

二、DRBD资源属性

resource name

可以使用除空白字符外的任意ACSII表中的字符；

drbd设备

drbd的设备的访问路径，设备文件/dev/drbd#；

disk

各节点为组成此drbd设备所提供的块设备，通常是一个磁盘分区；

网络属性

节点间为了实现跨主机磁盘镜像而使用的网络配置；

注意：用户空间工具与drdb与内核中使用的模块版本要保持一致，只有在使用drbdadm工具时才会读取配置文件，对多个资源的公共配置，可以提取出来只配置一次，通常保存在Common中，此外还有global配置，这种配置跟资源本身没有关系的；

三、原理

需要知道，DRBD是工作在系统内核空间，而不是用户空间，它直接复制的是二进制数据，这是它速度快的根本原因。

其次，DRBD至少需要两个节点来工作，一主一次。

DRBD的文件同步过程和普通复制过程的不同：

DRBD在数据进入Buffer Cache时，先经过DRBD这一层，复制一份数据经过TCP/IP协议封装，发送到另一个节点上，另一个节点通过TCP/IP协议来接受复制过来的数据，同步到次节点的DRBD设备上。

DRBD底层设备支持

DRBD需要构建在底层设备之上，然后构建出一个块设备出来。对于用户来说，一个DRBD设备，就像是一块物理的磁盘，可以在上面内创建文件系统。DRBD所支持的底层设备有以下这些类：

1、一个磁盘，或者是磁盘的某一个分区；

2、一个soft raid 设备；

3、一个LVM的逻辑卷；

DRBD在远程传输上支持三种模式：

协议A:异步复制协议

只要主节点完成本地写操作就认为写操作完成，并且需要复制的数据包会被存放本地TCP发送缓存中，当发生failover故障，数据可能会丢失。

协议B：内存同步（半同步，semi-synchronous）复制协议

当本地磁盘的写己经完成，并且复制数据包己经到达对应从节点，此时主节点才认为磁盘写入己经完成，failover不会导致数据丢失，但当所有节点同时出现故障时，主节点上多数最新写入的数据可能丢失。

协议C:同步复制

只有在本地和远程磁盘都确定写入己完成后，主节点才会认为写入完成。这样可确保发生单点故障时不会导致任何数据丢失，如果发生数据丢失，那也只会在所有节点同时存在错误存储时才会发生这种情况

角色、模式
- 角色
  在drbd构造的集群中，资源具有角色的概念，分别为primary和secondary。
  所有设为primary的资源将不受限制进行读写操作。可以创建文件系统，可以使用裸设备,甚至直接io。所有设为secondary的设备中不能挂载，不能读写
- 模式
  
  drbd也有drbd mode：单主模型（主从），双主模型
  在单主模型下drbd可以使用任意的文件系统
  单在双主模型下只能使用集群文件系统，常用的开源的集群文件系统有:ocfs2和gfs2

四、安装DRDB

前提：

设置主机名（要与uname -n一致，主机名可解析）

时间同步

无密钥登录

本机环境

系统	主机名	ip地址
Centos6.5 x86_64	essun.node1.com	192.168.1.109
Centos6.5 x86_64	essun.node2.com	192.168.1.108

注：

内核模块程序包一定要跟你的系统内核保持一致，uname -r查看你的内核版本，内核模块的版本必须要严格对应，而用户空间的模块就不那么严格要求了，两个节点的时间也需要保持一致性；

第一步、下载与内核版本相应的内核模块与用户空间工具（地址：http://packages.atrpms.net/dist/el6/drbd/）

[root@essun download]# uname -r
2.6.32-431.el6.x86_64

[root@essun download]# ls
drbd-8.4.3-33.el6.x86_64.rpm  drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm

[root@essun download]# yum isntall drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm drbd-8.4.3-33.el6.x86_64.rpm -y

修改配置文件（/etc/drbd.d/global_common.conf）

[root@essun drbd.d]# vim global_common.conf

[root@essun drbd.d]# cat global_common.conf
global {

    usage-count no;
#  是否参加DRBD使用者统计.默认是yes

    # minor-count dialog-refresh disable-ip-verification
}

common {

    handlers {

        # These are EXAMPLE handlers only.

        # They may have severe implications,

        # like hard resetting the node under certain circumstances.

        # Be careful when chosing your poison.

        pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";

        pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";

        local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";

        # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";

        # split-brain "/usr/lib/drbd/notify-split-brain.sh root";

        # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";

        # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";

        # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;

    }

    startup {

        # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb

        #wfc-timeout 120;

                 #(等待另一个节点上线的超时时长)

       # degr-wfc-timeout 240;

                #(等待超时后做降级处理)

        #outdated-wfc-timeout

                #(过期的等待超时)

                # wait-after-sb

                #(脑裂之后等待多长时长)

    }

    options {

        # cpu-mask on-no-data-accessible

    }

    disk {

        # size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes

        on-io-error detach;

                # 同步错误的做法是分离

        # disk-drain md-flushes resync-rate resync-after al-extents

                # c-plan-ahead c-delay-target c-fill-target c-max-rate

                # c-min-rate disk-timeout

    }

    net {

        protocol C;

 # 使用协议C.表示收到远程主机的写入确认后,则认为写入完成.

        cram-hmac-alg "sha1";

        shared-secret "drdb.essun.drdb.org";

 # 设置主备机之间通信使用的信息算法.

        # protocol timeout max-epoch-size max-buffers unplug-watermark

        # connect-int ping-int sndbuf-size rcvbuf-size ko-count

        # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri

        # after-sb-1pri after-sb-2pri always-asbp rr-conflict

        # ping-timeout data-integrity-alg tcp-cork on-congestion

        # congestion-fill congestion-extents csums-alg verify-alg

        # use-rle

    }

    syncer {

        rate 1000M;
}

 # 设置主备节点同步时的网络速率最大值,单位是字节.

}

提供物理块设备（两个节点的分区号最好相同），不需要格式化

[root@essun drbd.d]# fdisk -cu /dev/s
scd0      sda1      sg0       shm/      snd/      stderr    stdout
sda       sda2      sg1       snapshot  sr0       stdin     systty

[root@essun drbd.d]# fdisk -cu /dev/sda

Command (m for help): p

Disk /dev/sda: 128.8 GB, 128849018880 bytes

255 heads, 63 sectors/track, 15665 cylinders, total 251658240 sectors
Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006c47d

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1   *        2048      411647      204800   83  Linux

/dev/sda2          411648   126240767    62914560   8e  Linux LVM

Command (m for help): n
Command action

   e   extended

   p   primary partition (1-4)
p
Partition number (1-4): 3
First sector (126240768-251658239, default 126240768):
Using default value 126240768
Last sector, +sectors or +size{K,M,G} (126240768-251658239, default 251658239): +10G

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

[root@essun drbd.d]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2

[root@essun drbd.d]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3

[root@essun drbd.d]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3

[root@essun drbd.d]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3

定义DRBD磁盘资源

[root@essun drbd.d]# vim dbresouce.res
resource dbresource {

        on essun.node1.com { #on说明在哪个节点上，跟uname -n保持一致

                device /dev/drbd0; #在磁盘上表现的drbd叫什么名；

                disk   /dev/sda3; #所使用的物理磁盘设备是哪个；

                address 192.168.1.109:7789; #在node1这个节点上监听的套接字,默认监听在7789端口上；

                meta-disk internal; #保存drbd源数据信息，表示就放在自己的磁盘区分上，也可以放在外部的磁盘上；

        }

        on essun.node2.com{

                device /dev/drbd0;

                disk /dev/sda3;

                address 192.168.1.108:7789;

                meta-disk internal;

        } 
}

将这两个文件拷贝到node1中的相同目录下

[root@essun drbd.d]# scp dbresouce.res global_common.conf essun.node1.com:/etc

/drbd.d/

dbresouce.res                               100%  259     0.3KB/s   00:00

global_common.conf                          100% 2044     2.0KB/s   00:00

[root@essun drbd.d]#

在各自的节点上初始化资源

[root@essun drbd.d]# drbdadm create-md dbresource
#初始化资源列表
Writing meta data...
initializing activity log
NOT initializing bitmap

lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory
New drbd meta data block successfully created.

lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory

[root@essun drbd.d]# service drbd start
#启动服务
Starting DRBD resources: [

     create res: dbresource

   prepare disk: dbresource

    adjust disk: dbresource

     adjust net: dbresource
]
.

[root@essun drbd.d]# drbd-overview
#查看drbd的状态

  0:dbresource/0  Connected Secondary/Secondary Inconsistent/Inconsistent C r-----

[root@essun drbd.d]# watch -n 1 'cat /proc/drbd'
#实时查看drbd的状态

Every 1.0s: cat /proc/drbd                                                     Thu May  1 08:48:23 2014

version: 8.4.3 (api:1/proto:86-101)

GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00

 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----

    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10485404

上面表示两个节点的drbd都是从的，这是因为我们并没有对其中任何一个提升主的。

将其中一个节点的DRBD资源提升为主资源

[root@essun drbd.d]# drbdadm primary --force dbresource
#将当前节点的drbd资源提升为主资源

[root@essun drbd.d]# drbd

[root@essun drbd.d]# drbd-overview

  0:dbresource/0  SyncSource Primary/Secondary UpToDate/Inconsistent C r---n-

    [>...................] sync'ed:  6.1% (9624/10236)M

初始化之后（角色己经发生改变了）

[root@essun drbd.d]# drbd-overview

  0:dbresource/0  Connected Primary/Secondary UpToDate/UpToDate C r-----

格式化DRBD

[root@essun drbd.d]# mke2fs -t ext4 /dev/drbd0
mke2fs 1.41.12 (17-May-2010)
Filesystem label=

OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2621351 blocks

131067 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560

80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:

    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done                      

Creating journal (32768 blocks): done

Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

[root@essun drbd.d]# mount /dev/drbd0 /mnt

[root@essun drbd.d]# cd mnt

-bash: cd: mnt: No such file or directory

[root@essun drbd.d]# cd /mnt

[root@essun mnt]# ls
lost+found

[root@essun mnt]# cp /etc/inittab .

[root@essun mnt]# ls
inittab  lost+found

[root@essun mnt]#

提升另一个节点的DRBD资源

[root@essun /]# umount /mnt
#卸载drbd
[root@essun /]# drbdadm secondary dbresource
# 降级DRBD资源
[root@essun /]# drbd-overview

  0:dbresource/0  Connected Secondary/Secondary UpToDate/UpToDate C r-----
#以下是在另一个节点操作
[root@essun ~]# drbdadm primary dbresource
[root@essun ~]# mount /dev/drbd0 /mnt
[root@essun ~]# cd /mnt
[root@essun mnt]# ls
inittab  lost+found

如果想让DRBD能够自动切换的话，就要借助于messages layer程序了。

首先在对DRBD降级，停止服务，禁止开机自动启动

[root@essun /]# umount /mnt

[root@essun /]# drbdadm secondary dbresource

[root@essun /]# drbd-overview

  0:dbresource/0  Connected Secondary/Secondary UpToDate/UpToDate C r-----

[root@essun /]# service drbd stop
Stopping all DRBD resources: .

[root@essun /]# chkconfig drbd off

[root@essun /]# chkconfig --list drbd
drbd            0:off   1:off   2:off   3:off   4:off   5:off   6:off

第二步、安装mariadb

[root@essun download]# tar xf mariadb-10.0.10-linux-x86_64.tar.gz

[root@essun download]# mv mariadb-10.0.10-linux-x86_64 /usr/local

[root@essun download]# cd /usr/local/

[root@essun local]# ln -s mariadb-10.0.10-linux-x86_64 mysql

[root@essun local]# groupadd -r mysql

[root@essun local]# useradd -r -g mysql mysql

[root@essun local]# chown -R mysql.mysql mari/

启动DRBD服务，提升主资源

[root@essun ~]# service drbd start
Starting DRBD resources: [

     create res: dbresource

   prepare disk: dbresource

    adjust disk: dbresource

     adjust net: dbresource
]
.

[root@essun local]# drbd-overview

  0:dbresource/0  Connected Secondary/Secondary UpToDate/UpToDate C r-----

[root@essun local]# drbd-overview

  0:dbresource/0  Connected Primary/Secondary UpToDate/UpToDate C r-----

[root@essun local]# mkdir /mariadb

[root@essun local]# mount /dev/drbd0 /mariadb/

[root@essun local]# cd /mariadb/

[root@essun mariadb]# mkdir {data,logs}

[root@essun mariadb]# ls
data   logs  lost+found

[root@essun /]# chown -R mysql.mysql /mariadb/

[root@essun /]# ll /mariadb/
total 24
drwxr-xr-x 2 mysql mysql  4096 May  1 11:09 data
drwxr-xr-x 2 mysql mysql  4096 May  1 11:09 logs
drwx------ 2 mysql mysql 16384 May  1 09:42 lost+found

注：

此目录在另一个节点上也要一样。

修改配置文件与添加启动脚本

[root@essun mari]# cp support-files/my-large.cnf /etc/cnf

[root@essun mari]# vim /etc/my.cnf

[root@essun mari]# grep -v "^#" /etc/my.cnf |grep -v "
[client]
port        = 3306

socket      = /tmp/mysql.sock
[mysqld]
port        = 3306

socket      = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 256M
max_allowed_packet = 1M
table_open_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M
thread_concurrency = 8

datadir=/mariadb/data

log-bin=/mariadb/logs/mysql-bin
binlog_format=mixed

server-id   = 1
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[myisamchk]
key_buffer_size = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout

[root@essun mari]# cp support-files/mysql.server /etc/rc.d/init.d/mari

[root@essun mari]# chmod +x /etc/rc.d/init.d/mari

[root@essun mari]# chkconfig --add mari

[root@essun mari]# chkconfig mari off

初始化数据库,并登录测试

[root@essun mari]# scripts/mysql_install_db --user=mysql --datadir=/mariadb/data/

Installing MariaDB/MySQL system tables in '/mariadb/data/' ...
140501 11:22:15 [Note] InnoDB: The InnoDB memory heap is disabled
140501 11:22:15 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140501 11:22:15 [Note] InnoDB: Compressed tables use zlib 1.2.3
........省略中........

Please report any problems at http://mariadb.org/jira

The latest information about MariaDB is available at http://mariadb.org/.

You can find additional information about the MySQL part at:

http://dev.mysql.com

Support MariaDB development by buying support/new features from
SkySQL Ab. You can contact us about this at sales@skysql.com.
Alternatively consider joining our community based development effort:

http://mariadb.com/kb/en/contributing-to-the-mariadb-project/

[root@essun local]# service mari start
Starting MySQL........                                     [  OK  ]

[root@essun local]# /usr/local/mysql/bin/mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 4
Server version: 10.0.10-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |

| test               |
+--------------------+

4 rows in set (0.00 sec)
MariaDB [(none)]>

到另一个节点测试一下，先停止服务，卸载、降级

[root@essun local]# service mari stop
Shutting down MySQL..                                      [  OK  ]

[root@essun local]# umount /mariadb/

[root@essun local]# drbdadm secondary dbresource

[root@essun local]# drbd-overview

  0:dbresource/0  Connected Secondary/Secondary UpToDate/UpToDate C r-----

[root@essun local]# scp /etc/my.cnf essun.node1.com:/etc/

my.cnf                                                               100% 4936     4.8KB/s   00:00

[root@essun local]# scp -p /etc/rc.d/init.d/mari essun.node1.com:/etc/rc.d/init.d/

mari                                                                 100%   11KB  11.4KB/s   00:00

[root@essun local]#
==============以下是另一个节点的操作

[root@essun ~]# mkdir /mariadb

[root@essun ~]# drbdadm primary dbresource

[root@essun ~]# drbd-overview

  0:dbresource/0  Connected Primary/Secondary UpToDate/UpToDate C r-----

[root@essun ~]# chown -R mysql.mysql /mariadb/

[root@essun ~]# mount /dev/drbd0 /mariadb/

[root@essun ~]# cd /mariadb/

[root@essun mariadb]# ll
total 24
drwxr-xr-x 5 mysq mysql  4096 May  1 11:29 data
drwxr-xr-x 2 mysq mysql  4096 May  1 11:29 logs
drwx------ 2 mysq mysql 16384 May  1 09:42 lost+found

[root@essun /]# service mari start
Starting MySQL.......                                      [  OK  ]

[root@essun /]# /usr/local/mysql/bin/mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 4
Server version: 10.0.10-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |

| test               |
+--------------------+

4 rows in set (0.03 sec)
MariaDB [(none)]>

两个节点都要停止服务

[root@essun /]# service mari stop
Shutting down MySQL..                                      [  OK  ]

[root@essun /]# chkconfig mari off

[root@essun /]# chkconfig --list mari
mari            0:off   1:off   2:off   3:off   4:off   5:off   6:off

[root@essun /]# service drbd stop
Stopping all DRBD resources:
.

[root@essun /]# chkconfig drbd off

[root@essun /]# chkconfig --list drbd
drbd            0:off   1:off   2:off   3:off   4:off   5:off   6:off

第三步、安装corosync、pacemaker

[root@essun download]# yum install corosync pacemaker -y

[root@essun download]# yum install crmsh-1.2.6-4.el6.x86_64.rpm pssh-2.3.1-2.el6.x86_64.rpm -y

修改配置文件，生成密钥

[root@essun /]# cd /etc/corosync/

[root@essun corosync]# ls
corosync.conf.example       service.d
corosync.conf.example.udpu  uidgid.d

[root@essun corosync]# cp corosync.conf.example corosync.conf

[root@essun corosync]# vim corosync.conf

[root@essun corosync]# cat corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {

    version: 2

    secauth: off

    threads: 0

    interface {

        ringnumber: 0

        bindnetaddr: 192.168.1.1

        mcastaddr: 226.94.1.1

        mcastport: 5405

        ttl: 1

    }
}
logging {

    fileline: off

    to_stderr: no

    to_logfile: yes

    to_syslog: no

    logfile: /var/log/cluster/corosync.log

    debug: off

    timestamp: on

    logger_subsys {

        subsys: AMF

        debug: off

    }
}
amf {

    mode: disabled
}
service {

        name:   pacemaker

        ver:    0
}
aisexce {

        user:   root

        group:  root
}

[root@essun corosync]# mv /dev/random /dev/x

[root@essun corosync]# mv /dev/urandom /dev/random

[root@essun corosync]# corosync-keygen
Corosync Cluster Engine Authentication key generator.

Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.

Writing corosync key to /etc/corosync/authkey.

[root@essun corosync]# mv /dev/random /dev/urandom

[root@essun corosync]# mv /dev/x /dev/random

将这两个文件复制到另一个节点上一份

[root@essun corosync]# scp authkey corosync.conf essun.node1.com:/etc/corosync/

authkey                100%  128     0.1KB/s   00:00

corosync.conf          100%  554     0.5KB/s   00:00

[root@essun corosync]#

启动服务（两个节点都要启动服务）

[root@essun corosync]# service corosync start
Starting Corosync Cluster Engine (corosync):            [  OK  ]

[root@essun corosync]# crm status
Last updated: Thu May  1 10:05:25 2014
Last change: Thu May  1 10:05:21 2014 via crmd on essun.node2.com
Stack: classic openais (with plugin)
Current DC: essun.node2.com - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured, 2 expected votes
0 Resources configured
Online: [ essun.node1.com essun.node2.com ]

[root@essun corosync]#

禁用stonith裁决，忽略投票规则

crm(live)configure# property stonith-enabled=false
crm(live)configure# property no-quorum-policy=ignore
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node essun.node1.com
node essun.node2.com

property $id="cib-bootstrap-options" \

    dc-version="1.1.10-14.el6_5.3-368c726" \

    cluster-infrastructure="classic openais (with plugin)" \

    expected-quorum-votes="2" \

    stonith-enabled="false" \

    no-quorum-policy="ignore"

定义VIP，mariadb服务

crm(live)configure# primitive vip ocf:heartbeat:IPaddr params ip=192.168.1.150 op monitor interval=20s timeout=20s on-fail=restart

crm(live)configure# verify

crm(live)configure# primitive myserver lsb:mari op monitor interval=20s timeout=20s on-fail=restart

crm(live)configure# verify

crm(live)configure# commit

crm(live)configure# show

crm(live)configure# show
node essun.node1.com \

    attributes standby="off"
node essun.node2.com
primitive myserver lsb:mari \

    op monitor interval="20s" timeout="20s" on-fail="restart"
primitive vip ocf:heartbeat:IPaddr \

    params ip="192.168.1.150" \

    op monitor interval="20s" timeout="20s" on-fail="restart"

property $id="cib-bootstrap-options" \

    dc-version="1.1.10-14.el6_5.3-368c726" \

    cluster-infrastructure="classic openais (with plugin)" \

    expected-quorum-votes="2" \

    stonith-enabled="false" \

    no-quorum-policy="ignore"

配置drdb集群资源

crm(live)configure# primitive dbdrbd ocf:linbit:drbd params drbd_resource=dbresource op monitor role=Master interval=50s timeout=30s op monitor role=Slave interval=60s timeout=30s op start timeout=240s interval=0 op stop timeout=100s interval=0

crm(live)configure# verify

crm(live)configure# commit

crm(live)configure# show
node essun.node1.com \

        attributes standby="off"
node essun.node2.com
node essun.node1.com \

        attributes standby="off"
node essun.node2.com
primitive dbdrbd ocf:linbit:drbd \

        params drbd_resource="dbresource" \

        op monitor role="Master" interval="50s" timeout="30s" \

        op monitor role="Slave" interval="60s" timeout="30s" \

        op start timeout="240s" interval="0" \

        op stop timeout="100s" interval="0"
primitive myserver lsb:mari \

        op monitor interval="20s" timeout="20s" on-fail="restart"
primitive vip ocf:heartbeat:IPaddr \

        params ip="192.168.1.150" \

        op monitor interval="20s" timeout="20s" on-fail="restart"

property $id="cib-bootstrap-options" \

        dc-version="1.1.10-14.el6_5.3-368c726" \

        cluster-infrastructure="classic openais (with plugin)" \

        expected-quorum-votes="2" \

        stonith-enabled="false" \

        no-quorum-policy="ignore"

drbd需要同时运行在两个节点上，但只能有一个节点（primary/secondary模型）是Master，而另一个节点为Slave；因此，它是一种比较特殊的集群资源，其资源类型为多态（Multi-state）clone类型，即主机节点有Master和Slave之分，且要求服务刚启动时两个节点都处于slave状态，一般来说，只要是定义克隆资源，首先其本身必须是主资源

crm(live)configure# ms ms_dbdrdb dbdrbd meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show ms_dbdrdb
ms ms_dbdrdb dbdrbd \

    meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"

到node1验证一下，看看是不是变成了主资源

[root@essun /]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:bf:c2:82 brd ff:ff:ff:ff:ff:ff

    inet 192.168.1.109/24 brd 255.255.255.255 scope global eth0

    inet 192.168.1.150/24 brd 255.255.255.255 scope global secondary eth0

    inet6 fe80::20c:29ff:febf:c282/64 scope link tentative dadfailed

       valid_lft forever preferred_lft forever

定义文件系统资源（以便挂载DRBD的主资源，此处挂载的是mariadb的数据目录)

crm(live)configure# primitive dbstore ocf:heartbeat:Filesystem params device=/dev/drbd0 directory=/mariadb fstype=ext4 op monitor interval40s timeout=40s op start timeout=60s interval=0 op stop timeout=60 interval=0

crm(live)configure# verify

crm(live)configure# commit

WARNING: CIB changed in the meantime: won't touch it!
Do you still want to commit? y

crm(live)configure# show dbstore
primitive dbstore ocf:heartbeat:Filesystem \

    params device="/dev/drbd0" directory="/mariadb" fstype="ext4" \

    op monitor timeout="40s" interval="0" interval40s \

    op start timeout="60s" interval="0" \

    op stop timeout="60" interval="0

结合drbd的特性，只能在drbd服务将某节点设置为Primary以后方可启动。因此，还需要为这两个资源建立排列约束和顺序约束。

colocation vip_with_myserver inf: vip myserver

crm(live)configure# colocation myserver_with_dbststore inf: myserver
dbstore

crm(live)configure# colocation dbstore_with_ms_dbdrbd inf: dbstore ms_dbdrdb:Master

crm(live)configure# verify

crm(live)configure# commit

crm(live)configure# order ms_dbdrbd_before_dbstore inf: ms_dbdrdb:promote dbstore:start

crm(live)configure# verify

crm(live)configure# order vip_before_myserver inf: vip myserver

crm(live)configure# verify

crm(live)configure# order dbstore_before_myserver inf: dbstore:start myserver:start

crm(live)configure# verify

crm(live)configure# commit

crm(live)# status
Last updated: Thu May  1 15:29:04 2014
Last change: Thu May  1 15:28:55 2014 via crmd on essun.node2.com
Stack: classic openais (with plugin)
Current DC: essun.node2.com - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured, 2 expected votes
5 Resources configured
Online: [ essun.node1.com essun.node2.com ]

 vip    (ocf::heartbeat:IPaddr):        Started essun.node1.com

 myserver       (lsb:mari):     Started essun.node1.com

 Master/Slave Set: ms_dbdrdb [dbdrbd]

     Masters: [ essun.node1.com ]

     Slaves: [ essun.node2.com ]

 dbstore        (ocf::heartbeat:Filesystem):    Started essun.node1.com

到节点node1上看一下maridb下是否己经挂载DRBD的主资源

[root@essun mariadb]# hostname
essun.node1.com

[root@essun mariadb]# ls
data  logs  lost+found

这己表示在DRBD己经挂载到node1的mariadb目下

第四步、测试高可用mariadb

授权测试用户

[root@essun /]# /usr/local/mysql/bin/mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 4
Server version: 10.0.10-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> grant all on *.* to root@'192.168.1.%' identified by 'mari';
Query OK, 0 rows affected (0.00 sec)

使用vip登录测试

[root@essun mariadb]# hostname
essun.node2.com

[root@essun mariadb]# /usr/local/mysql/bin/mysql -uroot -p -h192.168.1.150
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 6
Server version: 10.0.10-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |

| test               |
+--------------------+

4 rows in set (0.08 sec)

模拟节点node1故障

crm(live)node# standby essun.node1.com
[root@essun ~]# crm status

Last updated: Thu May  1 16:28:59 2014

Last change: Thu May  1 16:27:34 2014 via crmd on essun.node2.com

Stack: classic openais (with plugin)

Current DC: essun.node1.com - partition with quorum

Version: 1.1.10-14.el6_5.3-368c726

2 Nodes configured, 2 expected votes

5 Resources configured
Node essun.node1.com: standby
Online: [ essun.node2.com ]

 vip    (ocf::heartbeat:IPaddr):    Started essun.node2.com

 myserver   (lsb:mari): Started essun.node2.com

 Master/Slave Set: ms_dbdrdb [dbdrbd]

     Masters: [ essun.node2.com ]

     Stopped: [ essun.node1.com ]

 dbstore    (ocf::heartbeat:Filesystem):    Started essun.node2.com

再次使用vip登录测试

[root@essun mariadb]# hostname
essun.node1.com

[root@essun mariadb]# /usr/local/mysql/bin/mysql -uroot -p -h192.168.1.150
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 9
Server version: 10.0.10-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |

| test               |
+--------------------+

4 rows in set (1.28 sec)

本文转自 jinlinger 51CTO博客，原文链接：http://blog.51cto.com/essun/1405326，如需转载请自行联系原作者