1. 创建的 volumes 是否可以导出数据

查询当前池

[root@tt-ceph-048146 ~]# ceph osd  lspools
0 rbd,1 volumes,

查询 volumes 池中的卷

[root@tt-ceph-048144 ceph]# rbd -p volumes ls
volume-be34745e-165f-4fc6-86c4-6248173abc00

导出卷

[root@tt-ceph-048144 ceph]# rbd -p volumes export  be34745e-165f-4fc6-86c4-6248173abc00  /tmp/volume-test
2015-08-04 17:07:46.071512 7fcec6ada880 -1 librbd::ImageCtx: error finding header: (2) No such file or directory
rbd: error opening image be34745e-165f-4fc6-86c4-6248173abc00: (2) No such file or directory
[root@tt-ceph-048144 ceph]# rbd -p volumes export volume-be34745e-165f-4fc6-86c4-6248173abc00  /tmp/volume-test

导出的卷只能够用于 ceph 导出, 而不可以令 linux 直接执行读写

[root@tt-ceph-048144 ceph]#  dumpe2fs  /tmp/volume-test
dumpe2fs 1.42.9 (28-Dec-2013)
dumpe2fs: Bad magic number in super-block 当尝试打开 /tmp/volume-test 时
找不到有效的文件系统超级块.

2. vm 进行数据写入时候, 是否会对 mon 节点保持长连接, 是否会对 ceph osd 节点保持长连接

 
  经测试, 当 vm 云盘后, 对应物理机即会产生一常连接到  mon 节点,  只有在 vm 产生数据写时候, 才会与 osd 节点产生对应的 TCP 连接 
 

 
  MON 节点的常连接并不会全部都集中到其中的一台 MON 节点中, 常连接会比较平均地分布到各个 mon 节点中 
 

参考生产环境上的 osd 连接

第一个 mon 节点, 有 13 个常连接
[root@sh-ceph-128212 ~]# for ip in 10.198.128.212; do  ssh $ip  "netstat -nt | grep 6789" | awk '{print $5}' |  grep -E -v '10.198.128.212|10.198.128.213|10.198.128.214|10.198.128.215|10.198.128.216|10.198.128.217|10.198.128.218|10.198.128.219'   ; done
10.198.128.30:47557
10.198.128.35:53771
10.198.128.37:58485
10.198.128.32:33849
10.198.128.38:46272
10.198.128.32:33819
10.198.128.33:40363
10.198.128.31:45110
10.198.128.32:56079
10.198.128.30:59187
10.198.128.32:33836
10.198.128.30:34132
10.198.128.38:46288
 
第二个 mon 节点, 有 7 个常连接
[root@sh-ceph-128212 ~]# for ip in 10.198.128.214; do  ssh $ip  "netstat -nt | grep 6789" | awk '{print $5}' |  grep -E -v '10.198.128.212|10.198.128.213|10.198.128.214|10.198.128.215|10.198.128.216|10.198.128.217|10.198.128.218|10.198.128.219'   ; done
10.198.129.72:44896
10.198.128.38:47764
10.198.128.33:51101
10.198.128.38:56206
10.198.128.32:44929
10.198.128.31:37976
10.198.128.37:56534
 
第三个 mon 节点, 有 12 个常连接
[root@sh-ceph-128212 ~]# for ip in 10.198.128.216; do  ssh $ip  "netstat -nt | grep 6789" | awk '{print $5}' |  grep -E -v '10.198.128.212|10.198.128.213|10.198.128.214|10.198.128.215|10.198.128.216|10.198.128.217|10.198.128.218|10.198.128.219'   ; done
10.198.128.31:37590
10.198.128.32:49009
10.198.128.36:52632
10.198.128.36:52615
10.198.128.30:54464
10.198.128.32:45783
10.198.128.36:46733
10.198.128.35:34866
10.198.128.36:46793
10.198.128.37:54421
10.198.128.33:52135
10.198.128.36:46713

3. 当 vm 进行数据写入时候, 断开一个 mon 节点, 是否会对数据写入造成影响

当前测试 compute 192.168.209.106 mon 192.168.48.144, 192.168.48.146, 192.168.48.148

1.  当前只有 192.168.48.144 节点中出现与客户的常连接
2.  让 compute 节点中的 vm 进行大数据复制 (1gb 左右数据复制), 产生了磁盘 IO
3.  这个时候利用 kill 命令删除 192.168.48.144 的 mon 进程, 并利用 ps 命令进行验证
4.  当 mon 进程被杀死后, 192.168.209.106 与 192.168.48.148 创建了新的 mon 常连接
5.  数据复制没有被影响

mon 进程不会自动恢复, 需要手动对 mon 进行启动

[root@tt-ceph-048144 ceph]# ceph -s
  cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
  health HEALTH_WARN 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148
monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148
    osdmap e245: 30 osds: 30 up, 30 in
    pgmap v1015: 10304 pgs, 2 pools, 4660 MB data, 1183 objects
         46119 MB used, 854 GB / 899 GB avail
         10304 active+clean

重启启动了 mon 服务后, 恢复正常

[root@tt-ceph-048144 ceph]# ceph -s
  cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
  health HEALTH_OK
    monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e246: 30 osds: 30 up, 30 in
     pgmap v1016: 10304 pgs, 2 pools, 4660 MB data, 1183 objects
         46119 MB used, 854 GB / 899 GB avail
         10304 active+clean

4. 当 vm 进行数据写入时候, 断开一个 osd 节点, 是否会对数据写入造成影响

1. 在 vm 写入数据的过程中, 关闭一台 osd 节点
2. 在数据完成后, 验证数据完成性
3. 重启启动 osd 节点,  发现 ceph 集群会自动恢复
4. ceph 集群完整性没有被破坏

在进行数据复制过程中, 把其中一台的 osd 进程全部 kill 掉

[root@tt-ceph-048144 ceph]# ceph osd tree
2015-08-04 14:57:32.063320 7fccf41a9700 0 -- :/1007979 >> 192.168.48.144:6789/0 pipe(0x7fccf00298e0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7fccf0029b70).fault
# id weight type name up/down reweight
-1 30 root default
-2 5 host tt-ceph-048144
0 1 osd.0 up 1
1 1 osd.1 up 1
2 1 osd.2 up 1
3 1 osd.3 up 1
4 1 osd.4 up 1
-3 5 host tt-ceph-048145
5 1 osd.5 up 1
6 1 osd.6 up 1
7 1 osd.7 up 1
8 1 osd.8 up 1
9 1 osd.9 up 1
-4 5 host tt-ceph-048146
10 1 osd.10 up 1
11 1 osd.11 up 1
12 1 osd.12 up 1
13 1 osd.13 up 1
14 1 osd.14 up 1
-5 5 host tt-ceph-048147
15 1 osd.15 up 1
16 1 osd.16 up 1
17 1 osd.17 up 1
18 1 osd.18 up 1
19 1 osd.19 up 1
-6 5 host tt-ceph-048148
20 1 osd.20 up 1
21 1 osd.21 up 1
22 1 osd.22 up 1
23 1 osd.23 up 1
24 1 osd.24 up 1
-7 5 host tt-ceph-048149
25 1 osd.25 down 1
26 1 osd.26 down 1
27 1 osd.27 down 1
28 1 osd.28 down 1
29 1 osd.29 down 1

检查 ceph 集群, 发现发生故障

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 5176 pgs degraded; 5171 pgs stuck unclean; 5176 pgs undersized; recovery 471/2598 objects degraded (18.129%); 5/30 in osds are down; 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     osdmap e239: 30 osds: 25 up, 30 in
      pgmap v924: 10304 pgs, 2 pools, 3392 MB data, 866 objects
            42279 MB used, 858 GB / 899 GB avail
            471/2598 objects degraded (18.129%)
                5128 active+clean
                5176 active+undersized+degraded

重新启动 osd, 发现有数据恢复现象

[root@tt-ceph-048144 ceph]# ceph -s
2015-08-04 14:58:26.309245 7f7eb818b700  0 -- :/1008324 >> 192.168.48.144:6789/0 pipe(0x7f7eb40298e0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f7eb4029b70).fault
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 3089 pgs degraded; 3085 pgs stuck unclean; 3089 pgs undersized; recovery 325/2835 objects degraded (11.464%); 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     osdmap e245: 30 osds: 30 up, 30 in
      pgmap v941: 10304 pgs, 2 pools, 3708 MB data, 945 objects
            43958 MB used, 856 GB / 899 GB avail
            325/2835 objects degraded (11.464%)
                7215 active+clean
                3089 active+undersized+degraded
recovery io 12881 kB/s, 3 objects/s
  client io 3969 kB/s wr, 2 op/s

恢复后, 发现, 数据正常了

[root@tt-ceph-048144 ceph]# ceph -s
2015-08-04 14:58:44.340787 7fd72bd78700  0 -- :/1008446 >> 192.168.48.144:6789/0 pipe(0x7fd7300298e0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd730029b70).fault
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     osdmap e245: 30 osds: 30 up, 30 in
      pgmap v952: 10304 pgs, 2 pools, 3925 MB data, 999 objects
            45058 MB used, 855 GB / 899 GB avail
               10304 active+clean
  client io 13701 kB/s rd, 20372 kB/s wr, 119 op/s

重启一下故障的 mon 服务

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     osdmap e245: 30 osds: 30 up, 30 in
      pgmap v1015: 10304 pgs, 2 pools, 4660 MB data, 1183 objects
            46119 MB used, 854 GB / 899 GB avail
               10304 active+clean

第二次测试
数据复制正常

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_OK
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e248: 30 osds: 30 up, 30 in
      pgmap v1052: 10304 pgs, 2 pools, 4680 MB data, 1188 objects
            46095 MB used, 854 GB / 899 GB avail
               10304 active+clean
  client io 5402 kB/s rd, 3031 kB/s wr, 46 op/s

对其中一台电脑中所有的 osd 执行 kill 操作
查询发现, osd 出现了故障, GP 也出现非正常现象

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 5176 pgs degraded; 5155 pgs stuck unclean; 5176 pgs undersized; recovery 930/4533 objects degraded (20.516%); 5/30 in osds are down
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e250: 30 osds: 25 up, 30 in
      pgmap v1098: 10304 pgs, 2 pools, 5966 MB data, 1511 objects
            52202 MB used, 848 GB / 899 GB avail
            930/4533 objects degraded (20.516%)
                5128 active+clean
                5176 active+undersized+degraded
  client io 16592 kB/s rd, 4918 B/s wr, 134 op/s

在数据复制完成后, 重启启动所有 osd

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 147 pgs degraded; 234 pgs peering; 12 pgs recovering; 29 pgs recovery_wait; 73 pgs stuck unclean; recovery 467/4620 objects degraded (10.108%)
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e257: 30 osds: 28 up, 28 in
      pgmap v1146: 10304 pgs, 2 pools, 6080 MB data, 1540 objects
            51945 MB used, 758 GB / 809 GB avail
            467/4620 objects degraded (10.108%)
                   1 inactive
                 234 peering
                9922 active+clean
                 106 active+degraded
                  29 active+recovery_wait+degraded
                  12 active+recovering+degraded

再次查询, 数据自动恢复中

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 174 pgs degraded; 19 pgs recovering; 76 pgs recovery_wait; 93 pgs stuck unclean; recovery 586/4620 objects degraded (12.684%)
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e261: 30 osds: 30 up, 30 in
      pgmap v1156: 10304 pgs, 2 pools, 6080 MB data, 1540 objects
            53205 MB used, 847 GB / 899 GB avail
            586/4620 objects degraded (12.684%)
                  79 active+degraded
               10130 active+clean
                  76 active+recovery_wait+degraded
                  19 active+recovering+degraded
recovery io 379 MB/s, 97 objects/s
  client io 9401 kB/s wr, 4 op/s

再查询

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_OK
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e261: 30 osds: 30 up, 30 in
      pgmap v1162: 10304 pgs, 2 pools, 6080 MB data, 1540 objects
            53553 MB used, 847 GB / 899 GB avail
               10304 active+clean
recovery io 52337 kB/s, 12 objects/s

最终, 恢复完成

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_OK
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e261: 30 osds: 30 up, 30 in
      pgmap v1170: 10304 pgs, 2 pools, 6080 MB data, 1540 objects
            51335 MB used, 849 GB / 899 GB avail
               10304 active+clean

5. 新增 osd 节点进行扩容

当前集群磁盘状态 ( tt-ceph-048144 ~ tt-ceph-048149)

[root@tt-ceph-048144 ceph]# ceph df
GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED
    899G      724G         175G         19.50
POOLS:
    NAME        ID     USED       %USED     MAX AVAIL     OBJECTS
    rbd         0           0         0          235G           0
    volumes     1      48041M      5.22          235G       12166

新加入节点, 执行下面操作

1. 主机名同步, 时间同步, 安装 ceph 软件
2. 执行磁盘格式化,  创建 osd , 挂载对应磁盘

参考格式化脚本

#!/bin/bash
LANG=en_US
disk=`fdisk -l | grep ^Disk  | grep sectors | grep sd | grep -v sda | awk -F[:\ ]  '{print $2}' | sort`
yum install -y hdparm
for partition in  $disk
do
  dd if=/dev/zero of=$partition bs=1M count=100
  parted -s $partition mklabel msdos
  parted $partition mkpart primary xfs 1 100%
  hdparm -z "$partition"1
  mkfs.xfs -f -i size=512  "$partition"1
done

参考创建 osd 脚本

#!/bin/bash
LANG=en_US
num=30
diskpart=`fdisk -l  | grep Linux | grep -v sda | awk '{print $1}' | sort`
for partition in $diskpart
do
    ceph osd create
    mkdir /var/lib/ceph/osd/ceph-$num
    echo $partition  /var/lib/ceph/osd/ceph-$num   xfs defaults 0 0 >> /etc/fstab
    let num++
done
mount -a

参考磁盘挂载情况

[root@tt-ceph-048150 tmp]# mount | grep osd
/dev/sdb1 on /var/lib/ceph/osd/ceph-30 type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sdc1 on /var/lib/ceph/osd/ceph-31 type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sdd1 on /var/lib/ceph/osd/ceph-32 type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sde1 on /var/lib/ceph/osd/ceph-33 type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sdf1 on /var/lib/ceph/osd/ceph-34 type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sdg1 on /var/lib/ceph/osd/ceph-35 type xfs (rw,relatime,attr2,inode64,noquota)

参考 osd tree

[root@tt-ceph-048144 ceph]# ceph osd tree
# id    weight  type name       up/down reweight
-1      30      root default
-2      5               host tt-ceph-048144
0       1                       osd.0   up      1
1       1                       osd.1   up      1
2       1                       osd.2   up      1
3       1                       osd.3   up      1
4       1                       osd.4   up      1
-3      5               host tt-ceph-048145
5       1                       osd.5   up      1
6       1                       osd.6   up      1
7       1                       osd.7   up      1
8       1                       osd.8   up      1
9       1                       osd.9   up      1
-4      5               host tt-ceph-048146
10      1                       osd.10  up      1
11      1                       osd.11  up      1
12      1                       osd.12  up      1
13      1                       osd.13  up      1
14      1                       osd.14  up      1
-5      5               host tt-ceph-048147
15      1                       osd.15  up      1
16      1                       osd.16  up      1
17      1                       osd.17  up      1
18      1                       osd.18  up      1
19      1                       osd.19  up      1
-6      5               host tt-ceph-048148
20      1                       osd.20  up      1
21      1                       osd.21  up      1
22      1                       osd.22  up      1
23      1                       osd.23  up      1
24      1                       osd.24  up      1
-7      5               host tt-ceph-048149
25      1                       osd.25  up      1
26      1                       osd.26  up      1
27      1                       osd.27  up      1
28      1                       osd.28  up      1
29      1                       osd.29  up      1
30      0       osd.30  down    0
31      0       osd.31  down    0
32      0       osd.32  down    0
33      0       osd.33  down    0
34      0       osd.34  down    0
35      0       osd.35  down    0

3 初始化 osd

参考初始化脚本

#!/bin/bash
LANG=en_US
num=30
diskpart=`fdisk -l  | grep Linux | grep -v sda | awk '{print $1}' | sort`
for partition in $diskpart
do
   ceph-osd -i $num --mkfs --mkkey --osd-uuid 60391f4d-734f-425d-89b2-1fcb26c876c8
   let num++
done

4 授权 osd

参考授权脚本

#!/bin/bash
LANG=en_US
num=30
diskpart=`fdisk -l  | grep Linux | grep -v sda | awk '{print $1}' | sort`
for partition in $diskpart
do
    ceph auth add osd.$num osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-$num/keyring
    let num++
done

5. 修改 crush map

ceph osd crush add-bucket tt-ceph-048150 host
ceph osd crush move tt-ceph-048150 root=defaultfor num in `seq 30 35`
do
ceph osd crush add osd.$num 1.0 root=default host=tt-ceph-048150
done

参考修改后的 osd tree

[root@tt-ceph-048150 tmp]# ceph osd tree
# id    weight  type name       up/down reweight
-1      36      root default
-2      5               host tt-ceph-048144
0       1                       osd.0   up      1
1       1                       osd.1   up      1
2       1                       osd.2   up      1
3       1                       osd.3   up      1
4       1                       osd.4   up      1
-3      5               host tt-ceph-048145
5       1                       osd.5   up      1
6       1                       osd.6   up      1
7       1                       osd.7   up      1
8       1                       osd.8   up      1
9       1                       osd.9   up      1
-4      5               host tt-ceph-048146
10      1                       osd.10  up      1
11      1                       osd.11  up      1
12      1                       osd.12  up      1
13      1                       osd.13  up      1
14      1                       osd.14  up      1
-5      5               host tt-ceph-048147
15      1                       osd.15  up      1
16      1                       osd.16  up      1
17      1                       osd.17  up      1
18      1                       osd.18  up      1
19      1                       osd.19  up      1
-6      5               host tt-ceph-048148
20      1                       osd.20  up      1
21      1                       osd.21  up      1
22      1                       osd.22  up      1
23      1                       osd.23  up      1
24      1                       osd.24  up      1
-7      5               host tt-ceph-048149
25      1                       osd.25  up      1
26      1                       osd.26  up      1
27      1                       osd.27  up      1
28      1                       osd.28  up      1
29      1                       osd.29  up      1
-8      6               host tt-ceph-048150
30      1                       osd.30  down    0
31      1                       osd.31  down    0
32      1                       osd.32  down    0
33      1                       osd.33  down    0
34      1                       osd.34  down    0
35      1                       osd.35  down    0

6. 启动 osd

for num in `seq 30 35`
do
touch /var/lib/ceph/osd/ceph-$num/sysvinit
/etc/init.d/ceph  -a start osd.$num
done

在 osd 启动之后, ceph 集群会自动执行数据迁移及数据平衡

[root@tt-ceph-048144 ceph]# ceph -w
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 1637 pgs degraded; 45 pgs recovering; 1592 pgs recovery_wait; 494 pgs stuck unclean; recovery 9690/36498 objects degraded (26.549%); 1 mons down, quorum 0,1 tt-ceph-048144,tt-ceph-048146
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 28, quorum 0,1 tt-ceph-048144,tt-ceph-048146
     osdmap e287: 36 osds: 30 up, 30 in
      pgmap v7407: 10304 pgs, 2 pools, 48041 MB data, 12166 objects
            178 GB used, 720 GB / 899 GB avail
            9690/36498 objects degraded (26.549%)
                8667 active+clean
                1592 active+recovery_wait+degraded
                  45 active+recovering+degraded
recovery io 55052 kB/s, 13 objects/s

再参考

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 850 pgs degraded; 9 pgs peering; 48 pgs recovering; 800 pgs recovery_wait; 1 pgs stuck inactive; 261 pgs stuck unclean; recovery 5158/36498 objects degraded (14.132%)
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 36, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e291: 36 osds: 33 up, 33 in
      pgmap v7412: 10304 pgs, 2 pools, 48041 MB data, 12166 objects
            180 GB used, 718 GB / 899 GB avail
            5158/36498 objects degraded (14.132%)
                   2 active+degraded
                   9 peering
                9445 active+clean
                 800 active+recovery_wait+degraded
                  48 active+recovering+degraded
recovery io 272 MB/s, 69 objects/s

再次参考

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 2236 pgs degraded; 167 pgs peering; 18 pgs recovering; 1793 pgs recovery_wait; 104 pgs stuck inactive; 737 pgs stuck unclean; recovery 11298/36495 objects degraded (30.958%); 1 mons down, quorum 0,2 tt-ceph-048144,tt-ceph-048148
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 50, quorum 0,2 tt-ceph-048144,tt-ceph-048148
     osdmap e297: 36 osds: 35 up, 35 in
      pgmap v7419: 10304 pgs, 2 pools, 48037 MB data, 12165 objects
            181 GB used, 867 GB / 1049 GB avail
            11298/36495 objects degraded (30.958%)
                  73 inactive
                 143 degraded
                 282 active+degraded
                 166 peering
                7828 active+clean
                   1 remapped+peering
                1793 active+recovery_wait+degraded
                  18 active+recovering+degraded
recovery io 11214 kB/s, 2 objects/s

约莫几分钟后, 数据同步完成

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_OK
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 66, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e302: 36 osds: 36 up, 36 in
      pgmap v7460: 10304 pgs, 2 pools, 48041 MB data, 12166 objects
            187 GB used, 892 GB / 1079 GB avail
               10304 active+clean

参考之前容量

[root@tt-ceph-048144 ceph]# ceph df
GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED
    899G      724G         175G         19.50
POOLS:
    NAME        ID     USED       %USED     MAX AVAIL     OBJECTS
    rbd         0           0         0          235G           0
    volumes     1      48041M      5.22          235G       12166

参考扩容后容量

[root@tt-ceph-048150 tmp]# ceph df
GLOBAL:
    SIZE      AVAIL     RAW USED     %RAW USED
    1049G      867G         181G         17.34
POOLS:
    NAME        ID     USED       %USED     MAX AVAIL     OBJECTS
    rbd         0           0         0          287G           0
    volumes     1      48037M      4.47          287G       12165

参考新机器的磁盘容量

[root@tt-ceph-048150 /]# df  -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   36G  3.8G   32G  11% /
devtmpfs                 1.9G     0  1.9G   0% /dev
tmpfs                    1.9G     0  1.9G   0% /dev/shm
tmpfs                    1.9G  8.6M  1.9G   1% /run
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1                497M   96M  401M  20% /boot
/dev/sdb1                 30G  4.6G   26G  16% /var/lib/ceph/osd/ceph-30
/dev/sdc1                 30G  5.0G   26G  17% /var/lib/ceph/osd/ceph-31
/dev/sdd1                 30G  5.1G   25G  17% /var/lib/ceph/osd/ceph-32
/dev/sde1                 30G  5.0G   26G  17% /var/lib/ceph/osd/ceph-33
/dev/sdf1                 30G  4.7G   26G  16% /var/lib/ceph/osd/ceph-34
/dev/sdg1                 30G  5.0G   26G  17% /var/lib/ceph/osd/ceph-35

参考其他机器磁盘容量

[root@tt-ceph-048144 ceph]# df -h
文件系统                 容量  已用  可用 已用% 挂载点
/dev/mapper/centos-root   36G   12G   25G   32% /
devtmpfs                 1.9G     0  1.9G    0% /dev
tmpfs                    1.9G     0  1.9G    0% /dev/shm
tmpfs                    1.9G  8.7M  1.9G    1% /run
tmpfs                    1.9G     0  1.9G    0% /sys/fs/cgroup
/dev/sdd1                 30G  4.8G   26G   16% /var/lib/ceph/osd/ceph-2
/dev/sdf1                 30G  4.9G   26G   17% /var/lib/ceph/osd/ceph-4
/dev/sdb1                 30G  5.1G   25G   17% /var/lib/ceph/osd/ceph-0
/dev/sde1                 30G  4.9G   26G   17% /var/lib/ceph/osd/ceph-3
/dev/sdc1                 30G  5.5G   25G   19% /var/lib/ceph/osd/ceph-1

6. osd 灾难性破坏与恢复

目标, 破坏一个 osd 对应的所有数据, 尝试恢复集群完整性

当前的 osd 环境正常,

[root@tt-ceph-048150 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   36G  3.8G   32G  11% /
devtmpfs                 1.9G     0  1.9G   0% /dev
tmpfs                    1.9G     0  1.9G   0% /dev/shm
tmpfs                    1.9G  8.7M  1.9G   1% /run
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1                497M   96M  401M  20% /boot
/dev/sdb1                 30G  4.6G   26G  16% /var/lib/ceph/osd/ceph-30
/dev/sdc1                 30G  5.0G   26G  17% /var/lib/ceph/osd/ceph-31
/dev/sdd1                 30G  5.0G   26G  17% /var/lib/ceph/osd/ceph-32
/dev/sde1                 30G  4.9G   26G  17% /var/lib/ceph/osd/ceph-33   <- 计划需要破坏的对象
/dev/sdf1                 30G  4.7G   26G  16% /var/lib/ceph/osd/ceph-34
/dev/sdg1                 30G  5.0G   26G  17% /var/lib/ceph/osd/ceph-35

[root@tt-ceph-048144 ~]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_OK
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 66, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e302: 36 osds: 36 up, 36 in
      pgmap v7570: 10304 pgs, 2 pools, 48041 MB data, 12166 objects
            178 GB used, 900 GB / 1079 GB avail
               10304 active+clean

破坏 osd 操作

[root@tt-ceph-048150 ~]# kill -9 14088
[root@tt-ceph-048150 ~]# umount /var/lib/ceph/osd/ceph-33

观察当前 osd 环境, 发现已经遭到破坏

[root@tt-ceph-048144 ~]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 805 pgs degraded; 779 pgs stuck unclean; 805 pgs undersized; recovery 1054/38058 objects degraded (2.769%); 1/36 in osds are down
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 66, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e304: 36 osds: 35 up, 36 in
      pgmap v7783: 10304 pgs, 2 pools, 50110 MB data, 12686 objects
            187 GB used, 891 GB / 1079 GB avail
            1054/38058 objects degraded (2.769%)
                9499 active+clean
                 805 active+undersized+degraded
  client io 8794 kB/s rd, 40401 kB/s wr, 90 op/s

[root@tt-ceph-048144 ~]#  ceph osd tree 
-8      6               host tt-ceph-048150
30      1                       osd.30  up      1
31      1                       osd.31  up      1
32      1                       osd.32  up      1
33      1                       osd.33  down    0
34      1                       osd.34  up      1
35      1                       osd.35  up      1

恢复 osd 过程

初始化磁盘

[root@tt-ceph-048150 ~]# dd if=/dev/zero of=/dev/sde1 bs=1M count=20
20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 0.0192865 s, 1.1 GB/s
[root@tt-ceph-048150 ~]# mkfs.xfs /dev/sde1
meta-data=/dev/sde1              isize=256    agcount=4, agsize=1966016 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0
data     =                       bsize=4096   blocks=7864064, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=3839, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
 
[root@tt-ceph-048150 ~]# mount /dev/sde1 /var/lib/ceph/osd/ceph-33

初始化 osd

[root@tt-ceph-048150 ~]#  ceph-osd -i 33 --mkfs --mkkey --osd-uuid 60391f4d-734f-425d-89b2-1fcb26c876c8
[root@tt-ceph-048150 ~]# ls /var/lib/ceph/osd/ceph-33/
ceph_fsid current fsid journal keyring magic ready store_version superblock whoami

授权会在初始化时候自动完成检查授权 key 是否正确

[root@tt-ceph-048144 ~]# ceph auth list
osd.33
    key: AQBC1sFVSGx/JhAASmGkmXgJKXt7yiHaNPNlHQ==
    caps: [mon] allow profile osd
    caps: [osd] allow *

检查 osd 初始化后的 keyring

[root@tt-ceph-048150 ~]# cat /var/lib/ceph/osd/ceph-33/keyring
[osd.33]
key = AQDPD8NVMMQLNxAAPtYAXbqycCekTVwvavuUSA==

注意: 加入 key 不对, 则需要手动修改, 否则会遇到下面的报错

[root@tt-ceph-048150 ~]# /etc/init.d/ceph start osd.33
=== osd.33 ===
Error connecting to cluster: PermissionError
2015-08-06 15:48:32.869373 7fa4cee30700 0 librados: osd.33 authentication error (1) Operation not permitted
failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.33 --keyring=/var/lib/ceph/osd/ceph-33/keyring osd crush create-or-move -- 33 0.03 host=tt-ceph-048150 root=default'

启动 osd

[root@tt-ceph-048150 ~]# touch /var/lib/ceph/osd/ceph-33/sysvinit
[root@tt-ceph-048150 ~]# /etc/init.d/ceph  start osd.33
=== osd.33 ===
create-or-move updated item name 'osd.33' weight 0.03 at location {host=tt-ceph-048150,root=default} to crush map
Starting Ceph osd.33 on tt-ceph-048150...
Running as unit run-19821.service.

参考 ceph 集群监控

[root@tt-ceph-048144 ~]# ceph osd tree
-8      6               host tt-ceph-048150
30      1                       osd.30  up      1
31      1                       osd.31  up      1
32      1                       osd.32  up      1
33      1                       osd.33  up      1
34      1                       osd.34  up      1
35      1                       osd.35  up      1

osd 启动后, 数据自动执行恢复

[root@tt-ceph-048150 ~]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 36 pgs degraded; 3 pgs recovering; 36 pgs stuck unclean; recovery 261/45192 objects degraded (0.578%)
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 72, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e308: 36 osds: 36 up, 36 in
      pgmap v8838: 10304 pgs, 2 pools, 59531 MB data, 15064 objects
            215 GB used, 863 GB / 1079 GB avail
            261/45192 objects degraded (0.578%)
                  33 active+degraded
               10268 active+clean
                   3 active+recovering+degraded
recovery io 191 MB/s, 48 objects/s

数据很快就恢复完成

[root@tt-ceph-048150 ~]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_OK
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 72, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e308: 36 osds: 36 up, 36 in
      pgmap v8856: 10304 pgs, 2 pools, 59531 MB data, 15064 objects
            215 GB used, 863 GB / 1079 GB avail
               10304 active+clean

7. 停电故障模拟

两个节点停电故障模拟

在集群环境正常状态, 为两个节点作停电处理

[root@tt-ceph-048144 /]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_OK
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 76, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e400: 36 osds: 36 up, 36 in
      pgmap v10445: 10304 pgs, 2 pools, 59531 MB data, 15064 objects
            212 GB used, 867 GB / 1079 GB avail
               10304 active+clean
  client io 29033 kB/s rd, 6084 B/s wr, 236 op/s

停电时, 涉及 1 个 mon 节点, 11 个 osd 节点

[root@tt-ceph-048144 /]# ceph osd tree
# id    weight  type name       up/down reweight
-1      36      root default
-2      5               host tt-ceph-048144
0       1                       osd.0   up      1
1       1                       osd.1   up      1
2       1                       osd.2   up      1
3       1                       osd.3   up      1
4       1                       osd.4   up      1
-3      5               host tt-ceph-048145
5       1                       osd.5   up      1
6       1                       osd.6   up      1
7       1                       osd.7   up      1
8       1                       osd.8   up      1
9       1                       osd.9   up      1
-4      5               host tt-ceph-048146
10      1                       osd.10  up      1
11      1                       osd.11  up      1
12      1                       osd.12  up      1
13      1                       osd.13  up      1
14      1                       osd.14  up      1
-5      5               host tt-ceph-048147
15      1                       osd.15  up      1
16      1                       osd.16  up      1
17      1                       osd.17  up      1
18      1                       osd.18  up      1
19      1                       osd.19  up      1
-6      5               host tt-ceph-048148
20      1                       osd.20  down    1
21      1                       osd.21  down    1
22      1                       osd.22  down    1
23      1                       osd.23  down    1
24      1                       osd.24  down    1
-7      5               host tt-ceph-048149
25      1                       osd.25  up      1
26      1                       osd.26  up      1
27      1                       osd.27  up      1
28      1                       osd.28  up      1
29      1                       osd.29  up      1
-8      6               host tt-ceph-048150
30      1                       osd.30  down    1
31      1                       osd.31  down    1
32      1                       osd.32  down    1
33      1                       osd.33  down    1
34      1                       osd.34  down    1
35      1                       osd.35  down    1

参见停电后的集群状态

[root@tt-ceph-048144 /]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 7614 pgs degraded; 7470 pgs stuck unclean; 7614 pgs undersized; recovery 13695/45744 objects degraded (29.938%); 11/36 in osds are down; 1 mons down, quorum 0,1 tt-ceph-048144,tt-ceph-048146
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 78, quorum 0,1 tt-ceph-048144,tt-ceph-048146
     osdmap e404: 36 osds: 25 up, 36 in
      pgmap v10492: 10304 pgs, 2 pools, 60265 MB data, 15248 objects
            216 GB used, 863 GB / 1079 GB avail
            13695/45744 objects degraded (29.938%)
                7614 active+undersized+degraded
                2690 active+clean
  client io 11370 kB/s rd, 4902 kB/s wr, 95 op/s

重新启动主机, 并重新启动 mon , osd 服务

再次参考 ceph 集群状态

[root@tt-ceph-048150 ~]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 4304 pgs degraded; 4247 pgs stuck unclean; 4304 pgs undersized; recovery 6765/47271 objects degraded (14.311%); 5/36 in osds are down
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 80, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e413: 36 osds: 31 up, 36 in
      pgmap v10584: 10304 pgs, 2 pools, 62283 MB data, 15757 objects
            223 GB used, 855 GB / 1079 GB avail
            6765/47271 objects degraded (14.311%)
                4304 active+undersized+degraded
                6000 active+clean
  client io 19230 kB/s rd, 155 op/s

[root@tt-ceph-048150 ~]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 281 pgs stale
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 80, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e422: 36 osds: 36 up, 36 in
      pgmap v10614: 10304 pgs, 2 pools, 62634 MB data, 15846 objects
            227 GB used, 852 GB / 1079 GB avail
                 281 stale+active+clean
               10023 active+clean
  client io 2283 kB/s rd, 11928 B/s wr, 20 op/s

故障自动修复, 数据复制没有被中断

[root@tt-ceph-048150 ~]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_OK
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 80, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e422: 36 osds: 36 up, 36 in
      pgmap v10621: 10304 pgs, 2 pools, 62766 MB data, 15879 objects
            227 GB used, 851 GB / 1079 GB avail
               10304 active+clean
  client io 12088 kB/s rd, 4543 kB/s wr, 101 op/s

三节点停电测试

[root@tt-ceph-048144 /]# ceph osd tree
^[[A# id        weight  type name       up/down reweight
-1      36      root default
-2      5               host tt-ceph-048144
0       1                       osd.0   up      1
1       1                       osd.1   up      1
2       1                       osd.2   up      1
3       1                       osd.3   up      1
4       1                       osd.4   up      1
-3      5               host tt-ceph-048145
5       1                       osd.5   up      1
6       1                       osd.6   up      1
7       1                       osd.7   up      1
8       1                       osd.8   up      1
9       1                       osd.9   up      1
-4      5               host tt-ceph-048146
10      1                       osd.10  up      1
11      1                       osd.11  up      1
12      1                       osd.12  up      1
13      1                       osd.13  up      1
14      1                       osd.14  up      1
-5      5               host tt-ceph-048147
15      1                       osd.15  down    1
16      1                       osd.16  down    1
17      1                       osd.17  down    1
18      1                       osd.18  down    1
19      1                       osd.19  down    1
-6      5               host tt-ceph-048148
20      1                       osd.20  down    1
21      1                       osd.21  down    1
22      1                       osd.22  down    1
23      1                       osd.23  down    1
24      1                       osd.24  down    1
-7      5               host tt-ceph-048149
25      1                       osd.25  up      1
26      1                       osd.26  up      1
27      1                       osd.27  up      1
28      1                       osd.28  up      1
29      1                       osd.29  up      1
-8      6               host tt-ceph-048150
30      1                       osd.30  down    1
31      1                       osd.31  down    1
32      1                       osd.32  down    1
33      1                       osd.33  down    1
34      1                       osd.34  down    1
35      1                       osd.35  down    1

参考停电期间的状态 (关闭 1 MON 节点, 16 OSD 节点)

[root@tt-ceph-048144 /]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 9232 pgs degraded; 321 pgs stale; 7797 pgs stuck unclean; 9232 pgs undersized; recovery 21510/49914 objects degraded (43.094%); 16/36 in osds are down; 1 mons down, quorum 0,1 tt-ceph-048144,tt-ceph-048146
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 82, quorum 0,1 tt-ceph-048144,tt-ceph-048146
     osdmap e426: 36 osds: 20 up, 36 in
      pgmap v10814: 10304 pgs, 2 pools, 65782 MB data, 16638 objects
            235 GB used, 844 GB / 1079 GB avail
            21510/49914 objects degraded (43.094%)
                8911 active+undersized+degraded
                1072 active+clean
                 321 stale+active+undersized+degraded
  client io 3291 kB/s rd, 2477 B/s wr, 27 op/s

重新启动三个节点, 启动 mon, osd 服务

[root@tt-ceph-048148 ~]# ceph osd tree
# id    weight  type name       up/down reweight
-1      36      root default
-2      5               host tt-ceph-048144
0       1                       osd.0   up      1
1       1                       osd.1   up      1
2       1                       osd.2   up      1
3       1                       osd.3   up      1
4       1                       osd.4   up      1
-3      5               host tt-ceph-048145
5       1                       osd.5   up      1
6       1                       osd.6   up      1
7       1                       osd.7   up      1
8       1                       osd.8   up      1
9       1                       osd.9   up      1
-4      5               host tt-ceph-048146
10      1                       osd.10  up      1
11      1                       osd.11  up      1
12      1                       osd.12  up      1
13      1                       osd.13  up      1
14      1                       osd.14  up      1
-5      5               host tt-ceph-048147
15      1                       osd.15  up      1
16      1                       osd.16  up      1
17      1                       osd.17  up      1
18      1                       osd.18  up      1
19      1                       osd.19  up      1
-6      5               host tt-ceph-048148
20      1                       osd.20  up      1
21      1                       osd.21  up      1
22      1                       osd.22  up      1
23      1                       osd.23  up      1
24      1                       osd.24  up      1
-7      5               host tt-ceph-048149
25      1                       osd.25  up      1
26      1                       osd.26  up      1
27      1                       osd.27  up      1
28      1                       osd.28  up      1
29      1                       osd.29  up      1
-8      6               host tt-ceph-048150
30      1                       osd.30  up      1
31      1                       osd.31  up      1
32      1                       osd.32  up      1
33      1                       osd.33  up      1
34      1                       osd.34  up      1
35      1                       osd.35  up      1

参考集群状态

[root@tt-ceph-048148 ~]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 2750 pgs degraded; 207 pgs down; 207 pgs peering; 207 pgs stuck inactive; 1822 pgs stuck unclean; 2749 pgs undersized; recovery 4525/49914 objects degraded (9.066%)
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 84, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e443: 36 osds: 36 up, 36 in
      pgmap v10843: 10304 pgs, 2 pools, 65782 MB data, 16638 objects
            232 GB used, 846 GB / 1079 GB avail
            4525/49914 objects degraded (9.066%)
                2749 active+undersized+degraded
                   1 active+degraded
                7347 active+clean
                 207 down+peering
recovery io 1116 MB/s, 283 objects/s
  client io 1247 MB/s rd, 2683 kB/s wr, 9946 op/s

集群会被自动修复成功

[root@tt-ceph-048148 ~]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_OK
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 84, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e443: 36 osds: 36 up, 36 in
      pgmap v10855: 10304 pgs, 2 pools, 65874 MB data, 16662 objects
            233 GB used, 845 GB / 1079 GB avail
               10304 active+clean
  client io 3240 kB/s rd, 26 op/s

整个 CEPH 集群停电测试

在所有 ceph 节点关闭后, 重启启动主机, 分别启动 MON OSD 服务后, 集群会自动恢复正常, 客户可继续访问, 读写云盘

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_OK
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 90, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e474: 36 osds: 36 up, 36 in
      pgmap v10961: 10304 pgs, 2 pools, 66864 MB data, 16911 objects
            239 GB used, 840 GB / 1079 GB avail
               10304 active+clean
  client io 2930 kB/s rd, 2929 B/s wr, 24 op/s

osd 故障测试

1. 创建的 volumes 是否可以导出数据

2. vm 进行数据写入时候, 是否会对 mon 节点保持长连接, 是否会对 ceph osd 节点保持长连接

3. 当 vm 进行数据写入时候, 断开一个 mon 节点, 是否会对数据写入造成影响

4. 当 vm 进行数据写入时候, 断开一个 osd 节点, 是否会对数据写入造成影响

5. 新增 osd 节点进行扩容

6. osd 灾难性破坏与恢复

7. 停电故障模拟

两个节点停电故障模拟

三节点停电测试

整个 CEPH 集群停电测试

热门文章

最新文章

相关课程

相关电子书

相关实验场景