第四章 创建不同类型的卷
学习完第三章关于GlusterFS的基础配置，进入到第四章创建不同类型的卷。不同类型的卷有着不同的作用，也有着不同的区别。好好理解各种卷的用途。
RHCA专栏地址：https://blog.csdn.net/qq_41765918/category_11532281.html
文章目录
练习与实验配置的volume是默认的分布式卷，glusterfs支持多种类型的volume，分别为4种基本类型卷，多种复合卷，分别如下：
distributed volume（分布式）
默认情况下，不指定类型，就是分布式volume。分布式volume，数据只保存一份，通过随机算法，选出一个birck保存。也就是说，一个brick down了之后，该volume就不可用了，架构图如下：
replicate volume（复制）
会对数据进行复制到不同的brick，复制份数，在创建该类型的volume时指定
在客户端看来，并不关心数据到底保存在哪里
该类型的磁盘使用率，只有一个brick的大小
gluster volume create vol2 replica 2\
node1:/brick/brick2/brick \
node2:/brick/brick2/brick
# 创建一个复制份数为2的复制类型volume
gluster volume status vol2
#会发现有一个叫:self-heal deamon，这个进程，是用来实现自愈，也就是复制。
架构图如下：
dispersed volume（分散）
类拟raid5，将数据进行分片，分布到brick中，其中一个brick用来作奇偶校验，允许一个brick down掉，比复制类型更省空间。需要的brick数量的计算方式如下：
-
6个brick，可设置冗余份数为2（4+2）
-
11个brick，可设置冗余份数为3（8+3）
-
12以上brick，可设置冗余份数为4（8+4）
比如：有6个brick，2个用来作奇偶校验，一份数据，会被分片成4份进行存储，允许2个brick down掉
gluster volume create vol4 disperse-data 4 redundancy 2 \
node1:/brick/brick4/brick \
node2:/brick/brick4/brick \
node3:/brick/brick4/brick \
node4:/brick/brick4/brick \
node5:/brick/brick4/brick \
node6:/brick/brick4/brick
注意1：如果想将同一个server上的多个brick加入同一个Volume，需要加上force选项
**注意2：**create创建参数时会有2个控制分片的参数：disperse和disperse-data，前者是控制总的分片数，包含奇偶校验块的数量，后者则为控制将原数据的分片数，不包含冗余块，故当6个brick，2个用来作奇偶校验，一份数据分片成4份进行存储这样的volume配置，使用disperse配置则为：
gluster volume create vol4 disperse 6 redundancy 2 \
node1:/brick/brick4/brick \
node2:/brick/brick4/brick \
node3:/brick/brick4/brick \
node4:/brick/brick4/brick \
node5:/brick/brick4/brick \
node6:/brick/brick4/brick
架构图：
作用：冗余并提高读I/O
stripe volume（条带）
类似RAID0，可以将一份数据平均分成多份存储进不同的brick，操作过程如下：
gluster volume create vol3 stripe 2 \
node1:/brick/brick3/brick \
node2:/brick/brick3/brick
架构图如下：
作用：提高I/O，缺点：一个brick故障，整个volume失效，且数据丢失
distribute replicate volume（分布复制）
既分布又复制，比如：2×3，2表示分布数，3表示复制数，共需要6个brick，以3为基础，一个数据复制3份，共分两组进行分布，第一个数据分布到第一组，并复制3份，第二个数据，分布到第二组，也复制3份
该类型磁盘使用率为复制数*单个brick容量
gluster volume create vol3 replica 2 \
node1:/brick/brick3/brick \
node2:/brick/brick3/brick \
node3:/brick/brick3/brick \
node4:/brick/brick3/brick
# 创建一个分布复制类型的volume，复制2份，2个分布
# 只需指定复制份数，分布数由brick总数/复制份数计算出来
架构图如下：
作用：扩容及冗余
distributed dipersed volume（分布分散）
比分散卷多一层分布调度，扩大容量，所以计算方法相对于分散卷是倍数计算的。如：2×(4+1)，表示，将数据分片成4份，1个作冗余，分布组为2个，也就是需要2×(4+1) =10个brick
gluster volume create vol5 disperse-data 4 redundancy 1 \
node1:/brick/brick5/brick node1:/brick/brick6/brick \
node2:/brick/brick5/brick node2:/brick/brick6/brick \
node3:/brick/brick5/brick node3:/brick/brick6/brick \
node4:/brick/brick5/brick node4:/brick/brick6/brick \
node5:/brick/brick5/brick node5:/brick/brick6/brick \
force
# 分布数量根据brick总数/一个分散组所需brick个数
架构图：
作用：冗余并扩容
删除volume
删除volume前需停止volume的服务，才能删除，否则会报错：
# gluster volume info vol1|grep Status
Status: Started
# gluster volume delete vol1
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: vol1: failed: Volume vol1 has been started.Volume needs to be stopped before deletion.
所以正确步骤为先停止服务，再删除卷
# gluster volume stop vol1
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: vol1: success
# gluster volume delete vol1
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: vol1: success
删除后，若需要重新利用起创建过的目录创建卷，需要删除里面的隐藏数据目录和目录的扩展属性：
1、 删除隐藏数据目录
这类目录是glusterfs启动volume时，对volume初始化生成的：
# tree -a /brick/brick1/brick1
/brick/brick1/brick1
├── .glusterfs
│ ├── 00
│ │ └── 00
│ │ ├── 00000000-0000-0000-0000-000000000001 -> ../../..
│ │ ├── 00000000-0000-0000-0000-000000000005 -> ../../00/00/00000000-0000-0000-0000-000000000001/.trashcan
│ │ └── 00000000-0000-0000-0000-000000000006 -> ../../00/00/00000000-0000-0000-0000-000000000005/internal_op
│ ├── brick1.db
│ ├── brick1.db-shm
│ ├── brick1.db-wal
│ ├── changelogs
│ │ ├── csnap
│ │ └── htime
│ ├── health_check
│ ├── indices
│ │ ├── dirty
│ │ └── xattrop
│ ├── landfill
│ ├── quanrantine
│ │ └── stub-00000000-0000-0000-0000-000000000008
│ └── unlink
└── .trashcan
└── internal_op
17 directories, 5 files
使用命令删除目录内文件：
rm -rf /brick/brick1/brick1/{.glusterfs,.trashcan}
2、 删除目录扩展属性
扩展属性在创建卷时就已添加，在卷启动时进一步添加更多的扩展属性标签，所以在重新创建前需删除干净扩展属性标签。
查看扩展属性标签：
# getfattr -d -m “.*” /brick/brick1/brick1/
getfattr: Removing leading ‘/’ from absolute path names
# file: brick/brick1/brick1/
security.selinux=“system_u:object_r:unlabeled_t:s0”
trusted.gfid=0sAAAAAAAAAAAAAAAAAAAAAQ==
trusted.glusterfs.dht=0sAAAAAQAAAAAAAAAAfg==
trusted.glusterfs.volume-id=0sf1W8jgybTceZGk4fsq4eew==
有标记引用的为glusterfs的标签，删除扩展属性标签：
# setfattr -x trusted.gfid /brick/brick1/brick1
# setfattr -x trusted.glusterfs.dht /brick/brick1/brick1
# setfattr -x trusted.glusterfs.volume-id /brick/brick1/brick1
当然，若文件夹内已无重要文件，需要重新引用时，直接删除即可，无需删除隐藏目录和扩展属性标签：
# rm -rf /brick/brick1/brick1
课本练习
[student@workstation ~]$ lab createvolumes setup
Setting up servera for lab exercise work:
• Testing if servera is reachable............................. PASS
• Testing if serverb is reachable............................. PASS
• Testing if serverc is reachable............................. PASS
• Testing if serverd is reachable............................. PASS
• Adding glusterfs to runtime firewall on servera............. SUCCESS
• Adding glusterfs to permanent firewall on servera........... SUCCESS
• Adding glusterfs to runtime firewall on serverb............. SUCCESS
• Adding glusterfs to permanent firewall on serverb........... SUCCESS
• Adding glusterfs to runtime firewall on serverc............. SUCCESS
• Adding glusterfs to permanent firewall on serverc........... SUCCESS
• Adding glusterfs to runtime firewall on serverd............. SUCCESS
• Adding glusterfs to permanent firewall on serverd........... SUCCESS
• Adding serverb.lab.example.com as peer...................... SUCCESS
• Adding serverc.lab.example.com as peer...................... SUCCESS
• Adding serverd.lab.example.com as peer...................... SUCCESS
• Ensuring thin LVM pool vg_bricks/thinpool exists on servera. SUCCESS
• Ensuring 2G LV in vg_bricks/thinpool on servera exists...... SUCCESS
• Ensuring xfs on /dev/vg_bricks/brick-a1 on servera.......... SUCCESS
• Verifying/adding /bricks/brick-a1 directory on servera...... SUCCESS
• Verifying/adding mount for /bricks/brick-a1 on servera...... SUCCESS
• Verifying/adding /bricks/brick-a1/brick directory on servera SUCCESS
• Ensuring fixed SELinux context for /bricks/brick-a1/brick on servera SUCCESS
• Restoring SElinux contexts on servera....................... SUCCESS
• Ensuring 2G LV in vg_bricks/thinpool on servera exists...... SUCCESS
• Ensuring xfs on /dev/vg_bricks/brick-a2 on servera.......... SUCCESS
• Verifying/adding /bricks/brick-a2 directory on servera...... SUCCESS
• Verifying/adding mount for /bricks/brick-a2 on servera...... SUCCESS
• Verifying/adding /bricks/brick-a2/brick directory on servera SUCCESS
• Ensuring fixed SELinux context for /bricks/brick-a2/brick on servera SUCCESS
• Restoring SElinux contexts on servera....................... SUCCESS
• Ensuring thin LVM pool vg_bricks/thinpool exists on serverb. SUCCESS
• Ensuring 2G LV in vg_bricks/thinpool on serverb exists...... SUCCESS
• Ensuring xfs on /dev/vg_bricks/brick-b1 on serverb.......... SUCCESS
• Verifying/adding /bricks/brick-b1 directory on serverb...... SUCCESS
• Verifying/adding mount for /bricks/brick-b1 on serverb...... SUCCESS
• Verifying/adding /bricks/brick-b1/brick directory on serverb SUCCESS
…………
1. 创建replicate volume
[root@servera ~]# gluster volume create replvol replica 2 servera:/bricks/brick-a1/brick/ serverb:/bricks/brick-b1/brick/
volume create: replvol: success: please start the volume to access data
[root@servera ~]# gluster volume start replvol
volume start: replvol: success
[root@servera ~]# gluster volume info replvol
Volume Name: replvol
Type: Replicate
Volume ID: d98665a7-e311-440c-a542-f96ed04ef277
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: servera:/bricks/brick-a1/brick
Brick2: serverb:/bricks/brick-b1/brick
Options Reconfigured:
performance.readdir-ahead: on
[root@servera ~]# showmount -e servera
Export list for servera:
/replvol *
[root@workstation ~]# mkdir /mnt/replvol
[root@workstation ~]# yum -y install glusterfs-fuse
[root@workstation ~]# mount -t glusterfs servera:/replvol /mnt/replvol/
[root@workstation ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 xfs 10G 3.0G 7.0G 31% /
devtmpfs devtmpfs 902M 0 902M 0% /dev
tmpfs tmpfs 920M 84K 920M 1% /dev/shm
tmpfs tmpfs 920M 17M 904M 2% /run
tmpfs tmpfs 920M 0 920M 0% /sys/fs/cgroup
tmpfs tmpfs 184M 16K 184M 1% /run/user/42
tmpfs tmpfs 184M 0 184M 0% /run/user/0
servera:/replvol fuse.glusterfs 2.0G 33M 2.0G 2% /mnt/replvol
2. 创建dispersed volume
[root@servera ~]# gluster volume create dispersevol disperse-data 4 redundancy 2 servera:/bricks/brick-a2/brick serverb:/bricks/brick-b2/brick serverc:/bricks/brick-c1/brick serverd:/bricks/brick-d1/brick serverc:/bricks/brick-c2/brick serverd:/bricks/brick-d2/brick force
volume create: dispersevol: success: please start the volume to access data
[root@servera ~]# gluster volume start dispersevol
volume start: dispersevol: success
[root@servera ~]# gluster volume info dispersevol
Volume Name: dispersevol
Type: Disperse
Volume ID: ba9ce31f-7cac-48ef-a4f1-079965f632ca
Status: Started
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: servera:/bricks/brick-a2/brick
Brick2: serverb:/bricks/brick-b2/brick
Brick3: serverc:/bricks/brick-c1/brick
Brick4: serverd:/bricks/brick-d1/brick
Brick5: serverc:/bricks/brick-c2/brick
Brick6: serverd:/bricks/brick-d2/brick
Options Reconfigured:
performance.readdir-ahead: on
[root@workstation ~]# mkdir /mnt/dispersevol
[root@workstation ~]# mount -t glusterfs servera:/dispersevol /mnt/dispersevol
[root@workstation ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 xfs 10G 3.0G 7.0G 31% /
devtmpfs devtmpfs 902M 0 902M 0% /dev
tmpfs tmpfs 920M 84K 920M 1% /dev/shm
tmpfs tmpfs 920M 17M 904M 2% /run
tmpfs tmpfs 920M 0 920M 0% /sys/fs/cgroup
tmpfs tmpfs 184M 16K 184M 1% /run/user/42
tmpfs tmpfs 184M 0 184M 0% /run/user/0
servera:/replvol fuse.glusterfs 2.0G 33M 2.0G 2% /mnt/replvol
servera:/dispersevol fuse.glusterfs 8.0G 131M 7.9G 2% /mnt/dispersevol
3. 复制文件到两个卷中，然后检查组成卷的单元的内容。
[root@workstation ~]# cp -R /boot /mnt/replvol/
[root@workstation ~]# cp -R /boot /mnt/dispersevol/
[root@servera ~]# ll -h /bricks/brick-a1/brick/boot/
total 108M
-rw-r--r--. 2 root root 124K Nov 26 14:46 config-3.10.0-327.el7.x86_64
drwxr-xr-x. 2 root root 37 Nov 26 14:46 grub
drwx------. 6 root root 104 Nov 26 14:46 grub2
-rw-r--r--. 2 root root 36M Nov 26 14:46 initramfs-0-rescue-4093bf66a4a4444886ac88feb9f56896.img
-rw-r--r--. 2 root root 35M Nov 26 14:46 initramfs-3.10.0-327.el7.x86_64.img
-rw-------. 2 root root 16M Nov 26 14:46 initramfs-3.10.0-327.el7.x86_64kdump.img
-rw-r--r--. 2 root root 9.8M Nov 26 14:46 initrd-plymouth.img
-rw-r--r--. 2 root root 247K Nov 26 14:46 symvers-3.10.0-327.el7.x86_64.gz
-rw-------. 2 root root 2.9M Nov 26 14:46 System.map-3.10.0-327.el7.x86_64
-rwxr-xr-x. 2 root root 5.0M Nov 26 14:46 vmlinuz-0-rescue-4093bf66a4a4444886ac88feb9f56896
-rwxr-xr-x. 2 root root 5.0M Nov 26 14:46 vmlinuz-3.10.0-327.el7.x86_64
[root@servera ~]# ll -h /bricks/brick-a2/brick/boot/
total 27M
-rw-r--r--. 2 root root 31K Nov 26 14:47 config-3.10.0-327.el7.x86_64
drwxr-xr-x. 2 root root 37 Nov 26 14:47 grub
drwx------. 6 root root 104 Nov 26 14:47 grub2
-rw-r--r--. 2 root root 8.8M Nov 26 14:47 initramfs-0-rescue-4093bf66a4a4444886ac88feb9f56896.img
-rw-r--r--. 2 root root 8.7M Nov 26 14:47 initramfs-3.10.0-327.el7.x86_64.img
-rw-------. 2 root root 3.8M Nov 26 14:47 initramfs-3.10.0-327.el7.x86_64kdump.img
-rw-r--r--. 2 root root 2.5M Nov 26 14:47 initrd-plymouth.img
-rw-r--r--. 2 root root 62K Nov 26 14:47 symvers-3.10.0-327.el7.x86_64.gz
-rw-------. 2 root root 724K Nov 26 14:47 System.map-3.10.0-327.el7.x86_64
-rwxr-xr-x. 2 root root 1.3M Nov 26 14:47 vmlinuz-0-rescue-4093bf66a4a4444886ac88feb9f56896
-rwxr-xr-x. 2 root root 1.3M Nov 26 14:47 vmlinuz-3.10.0-327.el7.x86_64
4. 评分脚本
[student@workstation ~]$ lab createvolumes grade
章节实验
[student@workstation ~]$ lab creatingvolumes-lab setup
Setting up servera for lab exercise work:
• Testing if servera is reachable............................. PASS
• Testing if serverb is reachable............................. PASS
• Testing if serverc is reachable............................. PASS
• Testing if serverd is reachable............................. PASS
• Testing if all nodes are present and accounted for.......... PASS
• Testing if vg_bricks/thinpool is available on servera....... PASS
• Testing if vg_bricks/thinpool is available on serverb....... PASS
• Testing if vg_bricks/thinpool is available on serverc....... PASS
• Testing if vg_bricks/thinpool is available on serverd....... PASS
Overall Pre-requisites check................................... PASS
Creating bricks for exercise
• Ensuring thin LVM pool vg_bricks/thinpool exists on servera. SUCCESS
• Ensuring 2G LV in vg_bricks/thinpool on servera exists...... SUCCESS
• Ensuring xfs on /dev/vg_bricks/brick-a1 on servera.......... SUCCESS
• Verifying/adding /bricks/brick-a1 directory on servera...... SUCCESS
• Verifying/adding mount for /bricks/brick-a1 on servera...... SUCCESS
• Verifying/adding /bricks/brick-a1/brick directory on servera SUCCESS
…………
1. 创建distribute replicate volume
[root@servera ~]# gluster volume create distreplvol replica 2 servera:/bricks/brick-a3/brick serverb:/bricks/brick-b3/brick serverc:/bricks/brick-c3/brick serverd:/bricks/brick-d3/brick
volume create: distreplvol: success: please start the volume to access data
[root@servera ~]# gluster volume start distreplvol
volume start: distreplvol: success
[root@servera ~]# gluster volume info distreplvol
Volume Name: distreplvol
Type: Distributed-Replicate
Volume ID: 2911369a-9287-4c91-ab3f-ac56ef504052
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: servera:/bricks/brick-a3/brick
Brick2: serverb:/bricks/brick-b3/brick
Brick3: serverc:/bricks/brick-c3/brick
Brick4: serverd:/bricks/brick-d3/brick
Options Reconfigured:
performance.readdir-ahead: on
2. 创建distributed dipersed volume
[root@servera ~]# cat server.sh
#!/bin/bash
for BRICKNUM in {
4..6}; do
for NODE in {
a..d}; do
echo server${NODE}:/bricks/brick-${NODE}${BRICKNUM}/brick
done
done
[root@servera ~]# sh server.sh > brick.txt
[root@servera ~]# echo $(</root/brick.txt)
servera:/bricks/brick-a4/brick serverb:/bricks/brick-b4/brick serverc:/bricks/brick-c4/brick serverd:/bricks/brick-d4/brick servera:/bricks/brick-a5/brick serverb:/bricks/brick-b5/brick serverc:/bricks/brick-c5/brick serverd:/bricks/brick-d5/brick servera:/bricks/brick-a6/brick serverb:/bricks/brick-b6/brick serverc:/bricks/brick-c6/brick serverd:/bricks/brick-d6/brick
[root@servera ~]# gluster volume create distdispvol disperse-data 4 redundancy 2 $(</root/brick.txt) force
volume create: distdispvol: success: please start the volume to access data
[root@servera ~]# gluster volume start distdispvol
volume start: distdispvol: success
[root@servera ~]# gluster volume info distdispvol
Volume Name: distdispvol
Type: Distributed-Disperse
Volume ID: 7ed166a5-cb55-47c7-8f5a-b0b8ddcc52fc
Status: Started
Number of Bricks: 2 x (4 + 2) = 12
Transport-type: tcp
Bricks:
Brick1: servera:/bricks/brick-a4/brick
Brick2: serverb:/bricks/brick-b4/brick
Brick3: serverc:/bricks/brick-c4/brick
Brick4: serverd:/bricks/brick-d4/brick
Brick5: servera:/bricks/brick-a5/brick
Brick6: serverb:/bricks/brick-b5/brick
Brick7: serverc:/bricks/brick-c5/brick
Brick8: serverd:/bricks/brick-d5/brick
Brick9: servera:/bricks/brick-a6/brick
Brick10: serverb:/bricks/brick-b6/brick
Brick11: serverc:/bricks/brick-c6/brick
Brick12: serverd:/bricks/brick-d6/brick
Options Reconfigured:
performance.readdir-ahead: on
3. 评分脚本
[student@workstation ~]$ lab creatingvolumes-lab grade
4. 重置环境
reset workstation,servera,serverb,serverc,serverd
总结
-
讲解各种卷的用途与创建。
-
删除已创建的卷需要删除隐藏数据目录和目录扩展属性。
以上就是【金鱼哥】的分享。希望能对看到此文章的小伙伴有所帮助。
如果这篇【文章】有帮助到你，希望可以给【金鱼哥】点个赞👍，创作不易，相比官方的陈述，我更喜欢用【通俗易懂】的文笔去讲解每一个知识点，如果有对【运维技术】感兴趣，也欢迎关注❤️❤️❤️ 【金鱼哥】❤️❤️❤️，我将会给你带来巨大的【收获与惊喜】💕💕！
