作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.准备工作
1>.实验环境说明
本次试验我打算采用4个分区设备来做RAID 5，如下图所示。
一般情况下使用3块设备就可以做RAID 5,但我之所以使用4块设备，是想用第4块设备充当空闲硬盘，当RAID 5中某一块设备损坏时，它会立即顶上去。
温馨提示：
由于我们使用的是分区来模拟RAID实战(即软RAID)，因此在分区时务必指定Id类型为"fd"(即"Linux raid auto")
生产环境中我们很少用磁盘分区来做RAID，而是直接使用整块磁盘来做RAID阵列，而且会有专门的磁盘阵列卡。
2>.为/dev/sdb磁盘新增一个200G的分区
[root@yinzhengjie.com ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/sdb: 644.2 GB, 644245094400 bytes, 1258291200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x8c3b1f6e
Device Boot Start End Blocks Id System
/dev/sdb1 2048 209717247 104857600 fd Linux raid autodetect
Command (m for help): n
Partition type:
p primary (1 primary, 0 extended, 3 free)
e extended
Select (default p): p
Partition number (2-4, default 2):
First sector (209717248-1258291199, default 209717248):
Using default value 209717248
Last sector, +sectors or +size{K,M,G} (209717248-1258291199, default 1258291199): +200G
Partition 2 of type Linux and of size 200 GiB is set
Command (m for help): p
Disk /dev/sdb: 644.2 GB, 644245094400 bytes, 1258291200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x8c3b1f6e
Device Boot Start End Blocks Id System
/dev/sdb1 2048 209717247 104857600 fd Linux raid autodetect
/dev/sdb2 209717248 629147647 209715200 83 Linux
Command (m for help): t
Partition number (1,2, default 2): 2
Hex code (type L to list all codes): L
0 Empty 24 NEC DOS 81 Minix / old Lin bf Solaris
1 FAT12 27 Hidden NTFS Win 82 Linux swap / So c1 DRDOS/sec (FAT-
2 XENIX root 39 Plan 9 83 Linux c4 DRDOS/sec (FAT-
3 XENIX usr 3c PartitionMagic 84 OS/2 hidden C: c6 DRDOS/sec (FAT-
4 FAT16 <32M 40 Venix 80286 85 Linux extended c7 Syrinx
5 Extended 41 PPC PReP Boot 86 NTFS volume set da Non-FS data
6 FAT16 42 SFS 87 NTFS volume set db CP/M / CTOS / .
7 HPFS/NTFS/exFAT 4d QNX4.x 88 Linux plaintext de Dell Utility
8 AIX 4e QNX4.x 2nd part 8e Linux LVM df BootIt
9 AIX bootable 4f QNX4.x 3rd part 93 Amoeba e1 DOS access
a OS/2 Boot Manag 50 OnTrack DM 94 Amoeba BBT e3 DOS R/O
b W95 FAT32 51 OnTrack DM6 Aux 9f BSD/OS e4 SpeedStor
c W95 FAT32 (LBA) 52 CP/M a0 IBM Thinkpad hi eb BeOS fs
e W95 FAT16 (LBA) 53 OnTrack DM6 Aux a5 FreeBSD ee GPT
f W95 Ext'd (LBA) 54 OnTrackDM6 a6 OpenBSD ef EFI (FAT-12/16/
10 OPUS 55 EZ-Drive a7 NeXTSTEP f0 Linux/PA-RISC b
11 Hidden FAT12 56 Golden Bow a8 Darwin UFS f1 SpeedStor
12 Compaq diagnost 5c Priam Edisk a9 NetBSD f4 SpeedStor
14 Hidden FAT16 <3 61 SpeedStor ab Darwin boot f2 DOS secondary
16 Hidden FAT16 63 GNU HURD or Sys af HFS / HFS+ fb VMware VMFS
17 Hidden HPFS/NTF 64 Novell Netware b7 BSDI fs fc VMware VMKCORE
18 AST SmartSleep 65 Novell Netware b8 BSDI swap fd Linux raid auto
1b Hidden W95 FAT3 70 DiskSecure Mult bb Boot Wizard hid fe LANstep
1c Hidden W95 FAT3 75 PC/IX be Solaris boot ff BBT
1e Hidden W95 FAT1 80 Old Minix
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'
Command (m for help): p
Disk /dev/sdb: 644.2 GB, 644245094400 bytes, 1258291200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x8c3b1f6e
Device Boot Start End Blocks Id System
/dev/sdb1 2048 209717247 104857600 fd Linux raid autodetect
/dev/sdb2 209717248 629147647 209715200 fd Linux raid autodetect
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# fdisk /dev/sdb
3>.为/dev/sdc磁盘新增一个200G的分区
[root@yinzhengjie.com ~]# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/sdc: 429.5 GB, 429496729600 bytes, 838860800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x8c3b1f6e
Device Boot Start End Blocks Id System
/dev/sdc1 2048 209717247 104857600 fd Linux raid autodetect
Command (m for help): n
Partition type:
p primary (1 primary, 0 extended, 3 free)
e extended
Select (default p):
Using default response p
Partition number (2-4, default 2):
First sector (209717248-838860799, default 209717248):
Using default value 209717248
Last sector, +sectors or +size{K,M,G} (209717248-838860799, default 838860799): +200G
Partition 2 of type Linux and of size 200 GiB is set
Command (m for help): t
Partition number (1,2, default 2):
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'
Command (m for help): p
Disk /dev/sdc: 429.5 GB, 429496729600 bytes, 838860800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x8c3b1f6e
Device Boot Start End Blocks Id System
/dev/sdc1 2048 209717247 104857600 fd Linux raid autodetect
/dev/sdc2 209717248 629147647 209715200 fd Linux raid autodetect
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# fdisk /dev/sdc
4>.为/dev/sdd磁盘新增一个200G的分区
[root@yinzhengjie.com ~]# fdisk /dev/sdd
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xed751bb5.
Command (m for help): p
Disk /dev/sdd: 859.0 GB, 858993459200 bytes, 1677721600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xed751bb5
Device Boot Start End Blocks Id System
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-1677721599, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-1677721599, default 1677721599): +200G
Partition 1 of type Linux and of size 200 GiB is set
Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'
Command (m for help): p
Disk /dev/sdd: 859.0 GB, 858993459200 bytes, 1677721600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xed751bb5
Device Boot Start End Blocks Id System
/dev/sdd1 2048 419432447 209715200 fd Linux raid autodetect
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# fdisk /dev/sdd
5>.为/dev/sde磁盘新增一个200G的分区
[root@yinzhengjie.com ~]# fdisk /dev/sde
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x487f53ac.
Command (m for help): p
Disk /dev/sde: 1073.7 GB, 1073741824000 bytes, 2097152000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x487f53ac
Device Boot Start End Blocks Id System
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-2097151999, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151999, default 2097151999): +200G
Partition 1 of type Linux and of size 200 GiB is set
Command (m for help): P
Disk /dev/sde: 1073.7 GB, 1073741824000 bytes, 2097152000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x487f53ac
Device Boot Start End Blocks Id System
/dev/sde1 2048 419432447 209715200 83 Linux
Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'
Command (m for help): p
Disk /dev/sde: 1073.7 GB, 1073741824000 bytes, 2097152000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x487f53ac
Device Boot Start End Blocks Id System
/dev/sde1 2048 419432447 209715200 fd Linux raid autodetect
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# fdisk /dev/sde
二.创建RAID 5块设备实战案例
1>.创建一个RAID 5设备
[root@yinzhengjie.com ~]# cat /proc/mdstat
Personalities : [raid0]
md1 : active raid0 sdc1[1] sdb1[0]
209582080 blocks super 1.2 512k chunks
unused devices: <none>
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -C -a yes /dev/md2 -l 5 -n 3 -x 1 -c 1M /dev/sd{b2,c2,d1,e1} #"-C -a yes"表示创建一个RAID设备，名称为"/dev/md2"。"-x"表示备用的设备，"-c"表示组块的大小为1M，默认最后一块设备(/dev/sde1)为备用块设备。
mdadm: /dev/sdb2 appears to contain an ext2fs file system
size=209715200K mtime=Thu Jan 1 08:00:00 1970
Continue creating array? yes
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# cat /proc/mdstat #当创建RAID设备时，由于块设备较大，因此需要等待一段时间，此时我们就可以看到创建过程的进度，如下所示，可以看到当前的进度时6.2%。
Personalities : [raid0] [raid6] [raid5] [raid4]
md2 : active raid5 sdd1[4] sde1[3](S) sdc2[1] sdb2[0]
419166208 blocks super 1.2 level 5, 1024k chunk, algorithm 2 [3/2] [UU_]
[=>...................] recovery = 6.2% (13109760/209583104) finish=7.2min speed=452060K/sec
bitmap: 0/2 pages [0KB], 65536KB chunk
md1 : active raid0 sdc1[1] sdb1[0]
209582080 blocks super 1.2 512k chunks
unused devices: <none>
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# cat /proc/mdstat #目前已经创建完毕啦~之前的进度信息也没啦！
Personalities : [raid0] [raid6] [raid5] [raid4]
md2 : active raid5 sdd1[4] sde1[3](S) sdc2[1] sdb2[0]
419166208 blocks super 1.2 level 5, 1024k chunk, algorithm 2 [3/3] [UUU]
bitmap: 0/2 pages [0KB], 65536KB chunk
md1 : active raid0 sdc1[1] sdb1[0]
209582080 blocks super 1.2 512k chunks
unused devices: <none>
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -C -a yes /dev/md2 -l 5 -n 3 -x 1 -c 1M /dev/sd{b2,c2,d1,e1} #"-C -a yes"表示创建一个RAID设备，名称为"/dev/md2"。"-x"表示备用的设备，"-c"表示组块的大小为1M，默认最后一块设备(/dev/sde1)为备用块设备。
2>.查看创建设备的详细信息
[root@yinzhengjie.com ~]# mdadm -D /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Fri Aug 21 03:27:31 2020
Raid Level : raid5
Array Size : 419166208 (399.75 GiB 429.23 GB)
Used Dev Size : 209583104 (199.87 GiB 214.61 GB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Aug 21 03:35:01 2020
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 1024K
Consistency Policy : bitmap
Name : yinzhengjie.com:2 (local to host yinzhengjie.com)
UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f
Events : 93
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
1 8 34 1 active sync /dev/sdc2
4 8 49 2 active sync /dev/sdd1
3 8 65 - spare /dev/sde1
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2
3>.对RAID 5设备进行格式化，指定为XFS文件系统
[root@yinzhengjie.com ~]# blkid
/dev/sdc1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="78244c8b-f961-577f-9d32-af6c474e9f2b" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member"
/dev/sdc2: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="142f3a98-d654-1abd-00bd-a50a66fec7fc" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sdd1: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="f1699604-e5f0-f054-31f6-a56f90d452e3" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sdb1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="595fb659-a792-aaac-3a2d-800b3e0795ac" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member"
/dev/sdb2: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="5a6802d5-93af-680d-dca1-64d02c5b615b" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sde1: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="e6b567f9-0b34-3acd-6d7d-39540d51878f" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sda1: PARTUUID="8f500d9c-87b9-421d-8ce0-8c385b73bf08"
/dev/sda2: UUID="1865a93f-6113-4097-89dc-8c4ea5fdf68c" TYPE="xfs" PARTUUID="e85154fa-bbc8-43c1-90f0-676000195a97"
/dev/sda3: UUID="lNmzXe-Whhd-NTtI-C55A-7m4x-CudG-1Q556q" TYPE="LVM2_member" PARTUUID="fa2e251a-d220-415f-acc7-da94b7a2d1b4"
/dev/sda4: PARTUUID="814caa31-3a7e-499d-8b7d-cc3017473ba8"
/dev/sr0: UUID="2018-11-25-21-21-31-00" LABEL="CentOS 7 x86_64" TYPE="iso9660" PTTYPE="dos"
/dev/mapper/centos-root: UUID="c0160ecb-06e4-497e-813c-53690b4a6109" TYPE="xfs"
/dev/md1: UUID="bc7d4ffa-439e-4544-b59e-58d10228569c" TYPE="xfs"
/dev/mapper/centos-yinzhengjie: UUID="6a2643fb-8d55-44cd-b41e-b5f2709ef855" TYPE="xfs"
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mkfs.xfs /dev/md2 #格式化RAID 5设备，将其格式化为XFS文件系统。
meta-data=/dev/md2 isize=512 agcount=16, agsize=6549248 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=104787968, imaxpct=25
= sunit=256 swidth=512 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=51168, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# blkid
/dev/sdc1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="78244c8b-f961-577f-9d32-af6c474e9f2b" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member"
/dev/sdc2: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="142f3a98-d654-1abd-00bd-a50a66fec7fc" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sdd1: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="f1699604-e5f0-f054-31f6-a56f90d452e3" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sdb1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="595fb659-a792-aaac-3a2d-800b3e0795ac" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member"
/dev/sdb2: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="5a6802d5-93af-680d-dca1-64d02c5b615b" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sde1: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="e6b567f9-0b34-3acd-6d7d-39540d51878f" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sda2: UUID="1865a93f-6113-4097-89dc-8c4ea5fdf68c" TYPE="xfs" PARTUUID="e85154fa-bbc8-43c1-90f0-676000195a97"
/dev/sda3: UUID="lNmzXe-Whhd-NTtI-C55A-7m4x-CudG-1Q556q" TYPE="LVM2_member" PARTUUID="fa2e251a-d220-415f-acc7-da94b7a2d1b4"
/dev/sr0: UUID="2018-11-25-21-21-31-00" LABEL="CentOS 7 x86_64" TYPE="iso9660" PTTYPE="dos"
/dev/mapper/centos-root: UUID="c0160ecb-06e4-497e-813c-53690b4a6109" TYPE="xfs"
/dev/md1: UUID="bc7d4ffa-439e-4544-b59e-58d10228569c" TYPE="xfs"
/dev/mapper/centos-yinzhengjie: UUID="6a2643fb-8d55-44cd-b41e-b5f2709ef855" TYPE="xfs"
/dev/sda1: PARTUUID="8f500d9c-87b9-421d-8ce0-8c385b73bf08"
/dev/sda4: PARTUUID="814caa31-3a7e-499d-8b7d-cc3017473ba8"
/dev/md2: UUID="eb94af80-9d27-4c8a-9c7b-1f54bb40048f" TYPE="xfs"
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mkfs.xfs /dev/md2 #格式化RAID 5设备，将其格式化为XFS文件系统。
4>.挂载设备
[root@yinzhengjie.com ~]# vim /etc/fstab
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mkdir /yinzhengjie/raid5
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# tail -1 /etc/fstab
UUID=eb94af80-9d27-4c8a-9c7b-1f54bb40048f /yinzhengjie/raid5 xfs defaults,noatime,nodiratime 0 0
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mount -a
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 500G 14G 486G 3% /
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 12M 2.0G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/sda2 1014M 133M 882M 14% /boot
/dev/mapper/centos-yinzhengjie 1.6T 2.4G 1.6T 1% /yinzhengjie
/dev/md1 200G 1.1G 199G 1% /yinzhengjie/raid
tmpfs 394M 0 394M 0% /run/user/0
/dev/md2 400G 33M 400G 1% /yinzhengjie/raid5 #为什么4块200GB的设备，但其大小仅有400G呢？
[root@yinzhengjie.com ~]#
5>.测试RAID 5设备的I/O性能
[root@yinzhengjie.com ~]# free -h
total used free shared buff/cache available
Mem: 3.8G 259M 3.4G 11M 143M 3.4G
Swap: 0B 0B 0B
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# dd if=/dev/zero of=/yinzhengjie/raid5/test.log bs=1M count=1024 #测试RAID 5设备的写性能
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.86494 s, 576 MB/s
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# free -h
total used free shared buff/cache available
Mem: 3.8G 263M 2.4G 11M 1.2G 3.3G
Swap: 0B 0B 0B
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# sync; echo 1 > /proc/sys/vm/drop_caches
[root@yinzhengjie.com ~]# sync; echo 2 > /proc/sys/vm/drop_caches
[root@yinzhengjie.com ~]# sync; echo 3 > /proc/sys/vm/drop_caches
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# free -h
total used free shared buff/cache available
Mem: 3.8G 261M 3.4G 11M 143M 3.4G
Swap: 0B 0B 0B
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# dd if=/yinzhengjie/raid5/test.log of=/dev/null #测试RAID 5设备的读性能
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB) copied, 2.85041 s, 377 MB/s
[root@yinzhengjie.com ~]#
三.测试RAID 5设备
1>.备份mdadm的配置信息
[root@yinzhengjie.com ~]# mdadm -Ds
ARRAY /dev/md/001 metadata=1.2 name=yinzhengjie.com:001 UUID=e7e1a4a7:f73b2990:a52b16b3:92be0561
ARRAY /dev/md2 metadata=1.2 spares=1 name=yinzhengjie.com:2 UUID=7bdb7eab:498a99bb:f69ee4ce:7d5c010f
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -Ds >/etc/mdadm.conf #将mdadm管理的RAID设备信息导出到一个配置文件中，如果人为损坏设备恢复时可以用到对应的UUID。
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# cat /etc/mdadm.conf
ARRAY /dev/md/001 metadata=1.2 name=yinzhengjie.com:001 UUID=e7e1a4a7:f73b2990:a52b16b3:92be0561
ARRAY /dev/md2 metadata=1.2 spares=1 name=yinzhengjie.com:2 UUID=7bdb7eab:498a99bb:f69ee4ce:7d5c010f
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -Ds >/etc/mdadm.conf #将mdadm管理的RAID设备信息导出到一个配置文件中，如果人为损坏设备恢复时可以用到对应的UUID。
2>.手动将RAID 5中的一块设备下线，观察备用设备是否会自动构建
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #观察操作之前的RAID 5设备信息
/dev/md2:
Version : 1.2
Creation Time : Fri Aug 21 03:27:31 2020
Raid Level : raid5
Array Size : 419166208 (399.75 GiB 429.23 GB)
Used Dev Size : 209583104 (199.87 GiB 214.61 GB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Aug 21 05:40:34 2020
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 1024K
Consistency Policy : bitmap
Name : yinzhengjie.com:2 (local to host yinzhengjie.com)
UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f
Events : 93
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
1 8 34 1 active sync /dev/sdc2
4 8 49 2 active sync /dev/sdd1
3 8 65 - spare /dev/sde1
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #观察操作之前的RAID 5设备信息
[root@yinzhengjie.com ~]# mdadm /dev/md2 -f /dev/sdc2 #手动下线"/dev/sdc2"这块设备，请仔细观察"RAID 5"设备的信息，你会发现空闲设备自动就顶替上来啦，如下图所示。
mdadm: set /dev/sdc2 faulty in /dev/md2
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Fri Aug 21 03:27:31 2020
Raid Level : raid5
Array Size : 419166208 (399.75 GiB 429.23 GB)
Used Dev Size : 209583104 (199.87 GiB 214.61 GB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Aug 21 05:51:07 2020
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 1
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 1024K
Consistency Policy : bitmap
Rebuild Status : 0% complete
Name : yinzhengjie.com:2 (local to host yinzhengjie.com)
UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f
Events : 95
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
3 8 65 1 spare rebuilding /dev/sde1
4 8 49 2 active sync /dev/sdd1
1 8 34 - faulty /dev/sdc2
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm /dev/md2 -f /dev/sdc2 #手动下线"/dev/sdc2"这块设备，请仔细观察"RAID 5"设备的信息，你会发现空闲设备自动就顶替上来啦，如下图所示。
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #当备用(Spare)设备替换了"faulty"状态的设备后，其状态"active"，中间状态为"spare rebuilding"(我测试环境的200G磁盘，需要5分钟左右才完成)
/dev/md2:
Version : 1.2
Creation Time : Fri Aug 21 03:27:31 2020
Raid Level : raid5
Array Size : 419166208 (399.75 GiB 429.23 GB)
Used Dev Size : 209583104 (199.87 GiB 214.61 GB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Aug 21 05:59:00 2020
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 1024K
Consistency Policy : bitmap
Name : yinzhengjie.com:2 (local to host yinzhengjie.com)
UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f
Events : 191
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
3 8 65 1 active sync /dev/sde1
4 8 49 2 active sync /dev/sdd1
1 8 34 - faulty /dev/sdc2
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #当备用(Spare)设备替换了"faulty"状态的设备后，其状态"active"，中间状态为"spare rebuilding"(我测试环境的200G磁盘，需要5分钟左右才完成)
3>.删除有错误的(faulty)设备
[root@yinzhengjie.com ~]# mdadm -D /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Fri Aug 21 03:27:31 2020
Raid Level : raid5
Array Size : 419166208 (399.75 GiB 429.23 GB)
Used Dev Size : 209583104 (199.87 GiB 214.61 GB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Aug 21 05:59:00 2020
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 1024K
Consistency Policy : bitmap
Name : yinzhengjie.com:2 (local to host yinzhengjie.com)
UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f
Events : 191
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
3 8 65 1 active sync /dev/sde1
4 8 49 2 active sync /dev/sdd1
1 8 34 - faulty /dev/sdc2
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #删除之前查看状态信息
[root@yinzhengjie.com ~]# mdadm /dev/md2 -r /dev/sdc2 #删除有错误的设备
mdadm: hot removed /dev/sdc2 from /dev/md2
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Fri Aug 21 03:27:31 2020
Raid Level : raid5
Array Size : 419166208 (399.75 GiB 429.23 GB)
Used Dev Size : 209583104 (199.87 GiB 214.61 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Aug 21 06:07:15 2020
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 1024K
Consistency Policy : bitmap
Name : yinzhengjie.com:2 (local to host yinzhengjie.com)
UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f
Events : 192
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
3 8 65 1 active sync /dev/sde1
4 8 49 2 active sync /dev/sdd1
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #删除之后再次查看状态信息
4>.往磁盘阵列中添加设备
[root@yinzhengjie.com ~]# mdadm -D /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Fri Aug 21 03:27:31 2020
Raid Level : raid5
Array Size : 419166208 (399.75 GiB 429.23 GB)
Used Dev Size : 209583104 (199.87 GiB 214.61 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Aug 21 06:07:15 2020
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 1024K
Consistency Policy : bitmap
Name : yinzhengjie.com:2 (local to host yinzhengjie.com)
UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f
Events : 192
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
3 8 65 1 active sync /dev/sde1
4 8 49 2 active sync /dev/sdd1
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #添加设备之前的状态
[root@yinzhengjie.com ~]# mdadm /dev/md2 -a /dev/sdc2 #我们将设备添加到RAID 5磁盘阵列中
mdadm: re-added /dev/sdc2
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Fri Aug 21 03:27:31 2020
Raid Level : raid5
Array Size : 419166208 (399.75 GiB 429.23 GB)
Used Dev Size : 209583104 (199.87 GiB 214.61 GB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Aug 21 06:10:38 2020
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 1024K
Consistency Policy : bitmap
Name : yinzhengjie.com:2 (local to host yinzhengjie.com)
UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f
Events : 193
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
3 8 65 1 active sync /dev/sde1
4 8 49 2 active sync /dev/sdd1
1 8 34 - spare /dev/sdc2
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #添加设备之后的状态
5>.验证数据是否存在
[root@yinzhengjie.com ~]# ll -h /yinzhengjie/raid5/ #当磁盘损坏时，我们发现文件依旧是可以正常访问的，原因是我们有备用的磁盘设备。
total 1.0G
-rw-r--r-- 1 root root 1.0G Aug 21 05:39 test.log
[root@yinzhengjie.com ~]#
6>.禁用RAID设备
[root@yinzhengjie.com ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 500G 14G 486G 3% /
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 12M 2.0G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/sda2 1014M 133M 882M 14% /boot
/dev/mapper/centos-yinzhengjie 1.6T 3.4G 1.6T 1% /yinzhengjie
/dev/md1 200G 1.1G 199G 1% /yinzhengjie/raid
tmpfs 394M 0 394M 0% /run/user/0
/dev/md2 400G 1.1G 399G 1% /yinzhengjie/raid5
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# umount /yinzhengjie/raid5/ #禁用RAID阵列之前，先要卸载对应的挂载点
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 500G 14G 486G 3% /
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 12M 2.0G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/sda2 1014M 133M 882M 14% /boot
/dev/mapper/centos-yinzhengjie 1.6T 3.4G 1.6T 1% /yinzhengjie
/dev/md1 200G 1.1G 199G 1% /yinzhengjie/raid
tmpfs 394M 0 394M 0% /run/user/0
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# umount /yinzhengjie/raid5/ #禁用RAID阵列之前，先要卸载对应的挂载点
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #查看磁盘阵列信息
/dev/md2:
Version : 1.2
Creation Time : Fri Aug 21 03:27:31 2020
Raid Level : raid5
Array Size : 419166208 (399.75 GiB 429.23 GB)
Used Dev Size : 209583104 (199.87 GiB 214.61 GB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Aug 21 06:25:31 2020
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 1024K
Consistency Policy : bitmap
Name : yinzhengjie.com:2 (local to host yinzhengjie.com)
UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f
Events : 193
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
3 8 65 1 active sync /dev/sde1
4 8 49 2 active sync /dev/sdd1
1 8 34 - spare /dev/sdc2
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -S /dev/md2 #禁用磁盘战列
mdadm: stopped /dev/md2
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #禁用后将无法查看磁盘阵列信息啦~
mdadm: cannot open /dev/md2: No such file or directory
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -S /dev/md2 #禁用磁盘阵列
7>.启用RAID设备
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #未启动磁盘阵列之前是无法查看相关信息的
mdadm: cannot open /dev/md2: No such file or directory
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -A /dev/md2 #启动磁盘阵列
mdadm: /dev/md2 has been started with 3 drives and 1 spare.
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #启用磁盘阵列后就可以查看该阵列的相关信息啦~
/dev/md2:
Version : 1.2
Creation Time : Fri Aug 21 03:27:31 2020
Raid Level : raid5
Array Size : 419166208 (399.75 GiB 429.23 GB)
Used Dev Size : 209583104 (199.87 GiB 214.61 GB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Aug 21 06:25:31 2020
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 1024K
Consistency Policy : bitmap
Name : yinzhengjie.com:2 (local to host yinzhengjie.com)
UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f
Events : 193
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
3 8 65 1 active sync /dev/sde1
4 8 49 2 active sync /dev/sdd1
1 8 34 - spare /dev/sdc2
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -A /dev/md2 #启动磁盘阵列
8>.清除磁盘的RAID信息
[root@yinzhengjie.com ~]# mdadm -D /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Fri Aug 21 03:27:31 2020
Raid Level : raid5
Array Size : 419166208 (399.75 GiB 429.23 GB)
Used Dev Size : 209583104 (199.87 GiB 214.61 GB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Aug 21 06:25:31 2020
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 1024K
Consistency Policy : bitmap
Name : yinzhengjie.com:2 (local to host yinzhengjie.com)
UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f
Events : 193
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
3 8 65 1 active sync /dev/sde1
4 8 49 2 active sync /dev/sdd1
1 8 34 - spare /dev/sdc2
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #查看磁盘阵列的RAID信息(注意观察UUID哟~)
[root@yinzhengjie.com ~]# mdadm -S /dev/md2 #禁用磁盘阵列
mdadm: stopped /dev/md2
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# blkid /dev/sd{b,c,d,e}* #查看"/dev/md2"磁盘阵列对应的分区文件，注意观察对应的UUID
/dev/sdb: PTTYPE="dos"
/dev/sdb1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="595fb659-a792-aaac-3a2d-800b3e0795ac" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member"
/dev/sdb2: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="5a6802d5-93af-680d-dca1-64d02c5b615b" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sdc: PTTYPE="dos"
/dev/sdc1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="78244c8b-f961-577f-9d32-af6c474e9f2b" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member"
/dev/sdc2: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="142f3a98-d654-1abd-00bd-a50a66fec7fc" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sdd: PTTYPE="dos"
/dev/sdd1: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="f1699604-e5f0-f054-31f6-a56f90d452e3" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sde: PTTYPE="dos"
/dev/sde1: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="e6b567f9-0b34-3acd-6d7d-39540d51878f" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm --zero-superblock /dev/sdb2 /dev/sdc2 /dev/sdd1 /dev/sde1 #我们将"/dev/md2"对应的RAID信息进行清空操作
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# blkid /dev/sd{b,c,d,e}* #再次观察各个分区对应的UUID，发现"/dev/md2"的UUID被成功删除啦~
/dev/sdb: PTTYPE="dos"
/dev/sdb1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="595fb659-a792-aaac-3a2d-800b3e0795ac" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member"
/dev/sdb2: LABEL="/mnt/sdb2" UUID="15028503-abd5-4bc7-9e35-2e745662d42d" TYPE="ext4"
/dev/sdc: PTTYPE="dos"
/dev/sdc1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="78244c8b-f961-577f-9d32-af6c474e9f2b" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member"
/dev/sdc2: UUID="2307c9c5-f7b1-4a2b-b9e9-46d89a9c7c81" TYPE="xfs"
/dev/sdd: PTTYPE="dos"
/dev/sde: PTTYPE="dos"
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm --zero-superblock /dev/sdb2 /dev/sdc2 /dev/sdd1 /dev/sde1 #我们将"/dev/md2"对应的RAID信息进行清空操作
