一、GlusterFS基础环境的介绍
1、关于GlusterFS文件系统和架构的介绍
http://jingyan.baidu.com/article/046a7b3ef65250f9c27fa9d9.html
2、实验的目的
a. 利用多台性能较低并且老旧的服务器，实现企业的云盘功能
b. GlusterFS服务端和客户端的部署和配置
c. 实现GlusterFS多节点的负载均衡功能
3、测试环境说明
操作系统：CentOS 6.7 X64
内核版本：2.6.32-573.el6.x86_64
软件版本：glusterfs 3.7.10
使用4台服务器创卷GlusterFS的DHT功能，客户端win10使用samba进行连接配置
二、GlusterFS服务端的配置（server01）
1、GlusterFS无中心化概念，很多关于GlusterFS的配置仅需要在其中一台主机设置
2、配置NTP服务器同步（这里也可以在crontab脚本里面，添加一个定时任务）
|
1
2
3
|
[root@server01 ~]
# ntpdate -u 10.203.10.20
18 Apr 14:16:15 ntpdate[2700]: adjust
time
server 10.203.10.20 offset 0.008930 sec
[root@server01 ~]
# hwclock -w
|
3、查看hosts表的记录
|
1
2
3
4
5
6
7
|
[root@server01 ~]
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.11server01
192.168.1.12server02
192.168.1.13server03
192.168.1.14server04
|
4、单独添加一块磁盘作为共享卷使用（这里也可以配置LVM）
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
|
[root@server01 ~]
# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x0ef88f22.
Changes will remain
in
memory only,
until
you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (
command
'c'
) and change display
units
to
sectors (
command
'u'
).
Command (m
for
help): p
Disk
/dev/sdb
: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors
/track
, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
Disk identifier: 0x0ef88f22
Device Boot Start End Blocks Id System
Command (m
for
help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-2610, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-2610, default 2610):
Using default value 2610
Command (m
for
help): w
The partition table has been altered!
Calling ioctl() to re-
read
partition table.
Syncing disks.
[root@server01 ~]
# partx /dev/sdb
# 1: 63- 41929649 ( 41929587 sectors, 21467 MB)
# 2: 0- -1 ( 0 sectors, 0 MB)
# 3: 0- -1 ( 0 sectors, 0 MB)
# 4: 0- -1 ( 0 sectors, 0 MB)
[root@server01 ~]
# fdisk -l |grep /dev/sdb
Disk
/dev/sdb
: 21.5 GB, 21474836480 bytes
/dev/sdb1
1 2610 20964793+ 83 Linux
[root@server01 ~]
# mkfs.ext4 /dev/sdb1
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS
type
: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1310720 inodes, 5241198 blocks
262059 blocks (5.00%) reserved
for
the super user
First data block=0
Maximum filesystem blocks=4294967296
160 block
groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000
Writing inode tables:
done
Creating journal (32768 blocks):
done
Writing superblocks and filesystem accounting information:
done
This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
|
5、创建挂载目录，并设置开机挂载
|
1
2
3
4
5
6
7
8
|
[root@server01 ~]
# mkdir -p /glusterfs-xfs-mount
[root@server01 ~]
# mount /dev/sdb1 /glusterfs-xfs-mount/
[root@server01 ~]
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3
193G 7.2G 176G 4% /
tmpfs 932M 0 932M 0%
/dev/shm
/dev/sda1
190M 41M 139M 23%
/boot
/dev/sdb1
20G 44M 19G 1%
/glusterfs-xfs-mount
|
6、修改自动挂载
|
1
|
[root@server01 ~]
# echo '/dev/sdb1 /glusterfs-xfs-mount xfs defaults 0 0' >> /etc/fstab
|
7、添加外部源
|
1
2
|
[root@server01 ~]
# cd /etc/yum.repos.d/
[root@server01 yum.repos.d]
# wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo
|
8、安装glusterfs服务端软件，并启动服务
|
1
2
3
4
5
6
7
8
9
10
11
12
|
[root@server01 yum.repos.d]
# yum -y install glusterfs-server
[root@server01 yum.repos.d]
# /etc/init.d/glusterd start
Starting glusterd: [ OK ]
[root@server01 yum.repos.d]
# chkconfig glusterd on
[root@server01 yum.repos.d]
# chkconfig --list glusterd
glusterd 0:off1:off2:on3:on4:on5:on6:off
[root@server01 yum.repos.d]
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3
193G 7.2G 176G 4% /
tmpfs 932M 0 932M 0%
/dev/shm
/dev/sda1
190M 41M 139M 23%
/boot
/dev/sdb1
20G 44M 19G 1%
/glusterfs-xfs-mount
|
9、添加集群对象（server02以及server03）
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
[root@server01 yum.repos.d]
# gluster peer status
Number of Peers: 0
[root@server01 yum.repos.d]
# gluster peer probe server02
peer probe: success.
[root@server01 yum.repos.d]
# gluster peer status
Number of Peers: 1
Hostname: server02
Uuid: c58d0715-32ff-4962-90d9-4275fa65793a
State: Peer
in
Cluster (Connected)
[root@server01 yum.repos.d]
# gluster peer probe server03
peer probe: success.
[root@server01 yum.repos.d]
# gluster peer status
Number of Peers: 2
Hostname: server02
Uuid: c58d0715-32ff-4962-90d9-4275fa65793a
State: Peer
in
Cluster (Connected)
Hostname: server03
Uuid: 5110d0af-fdd9-4c82-b716-991cf0601b53
State: Peer
in
Cluster (Connected)
|
10、创建Gluster Volume
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
[root@server01 yum.repos.d]
# gluster volume create dht-volume01 server01:/glusterfs-xfs-mount server02:/gluste
rfs-xfs-
mount
server03:
/glusterfs-xfs-mount
volume create: dht-volume01: failed: The brick server01:
/glusterfs-xfs-mount
is a
mount
point. Please create a
sub-directory under the
mount
point and use that as the brick directory. Or use
'force'
at the end of the com
mand
if
you want to override this behavior.
[root@server01 yum.repos.d]
# echo $?
1
[root@server01 yum.repos.d]
# gluster volume create dht-volume01 server01:/glusterfs-xfs-mount server02:/gluste
rfs-xfs-
mount
server03:
/glusterfs-xfs-mount
force
volume create: dht-volume01: success: please start the volume to access data
[root@server01 yum.repos.d]
# gluster volume start dht-volume01
volume start: dht-volume01: success
[root@server01 yum.repos.d]
# gluster volume status
Status of volume: dht-volume01
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick server01:
/glusterfs-xfs-mount
49152 0 Y 2948
Brick server02:
/glusterfs-xfs-mount
49152 0 Y 2910
Brick server03:
/glusterfs-xfs-mount
49152 0 Y 11966
NFS Server on localhost N
/A
N
/A
N N
/A
NFS Server on server02 N
/A
N
/A
N N
/A
NFS Server on server03 N
/A
N
/A
N N
/A
Task Status of Volume dht-volume01
------------------------------------------------------------------------------
There are no active volume tasks
|
11、测试写入一个512M的文件
|
1
2
3
4
5
6
7
|
[root@server01 yum.repos.d]
# cd /glusterfs-xfs-mount/
[root@server01 glusterfs-xfs-
mount
]
# dd if=/dev/zero of=test.img bs=1M count=512
512+0 records
in
512+0 records out
536870912 bytes (537 MB) copied, 5.20376 s, 103 MB
/s
[root@server01 glusterfs-xfs-
mount
]
# ls
lost+found
test
.img
|
三、GlusterFS服务端的配置（server02与server03上的配置类似）
1、配置时间同步，这里的ntp服务器IP地址为10.203.10.20
|
1
2
3
|
[root@server02 ~]
# ntpdate -u 10.203.10.20
18 Apr 14:27:58 ntpdate[2712]: adjust
time
server 10.203.10.20 offset -0.085282 sec
[root@server02 ~]
# hwclock -w
|
2、查看host表的信息
|
1
2
3
4
5
6
7
|
[root@server02 ~]
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.11server01
192.168.1.12server02
192.168.1.13server03
192.168.1.14server04
|
3、在本地设置一块单独的盘，组成GlusterFS卷的一部分
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
[root@server02 ~]
# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x927b5e72.
Changes will remain
in
memory only,
until
you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (
command
'c'
) and change display
units
to
sectors (
command
'u'
).
Command (m
for
help): p
Disk
/dev/sdb
: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors
/track
, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
Disk identifier: 0x927b5e72
Device Boot Start End Blocks Id System
Command (m
for
help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-2610, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-2610, default 2610):
Using default value 2610
Command (m
for
help): w
The partition table has been altered!
Calling ioctl() to re-
read
partition table.
Syncing disks.
|
4、更新磁盘分区表信息，使磁盘分区和格式化
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
[root@server02 ~]
# partx /dev/sdb
# 1: 63- 41929649 ( 41929587 sectors, 21467 MB)
# 2: 0- -1 ( 0 sectors, 0 MB)
# 3: 0- -1 ( 0 sectors, 0 MB)
# 4: 0- -1 ( 0 sectors, 0 MB)
[root@server02 ~]
# fdisk -l|grep /dev/sdb
Disk
/dev/sdb
: 21.5 GB, 21474836480 bytes
/dev/sdb1
1 2610 20964793+ 83 Linux
[root@server02 ~]
# mkfs.ext4 /dev/sdb1
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS
type
: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1310720 inodes, 5241198 blocks
262059 blocks (5.00%) reserved
for
the super user
First data block=0
Maximum filesystem blocks=4294967296
160 block
groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000
Writing inode tables:
done
Creating journal (32768 blocks):
done
Writing superblocks and filesystem accounting information:
done
This filesystem will be automatically checked every 21 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@server02 ~]
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3
193G 7.3G 176G 4% /
tmpfs 932M 0 932M 0%
/dev/shm
/dev/sda1
190M 41M 139M 23%
/boot
|
5、创建挂载目录，并配置挂载信息
|
1
2
3
4
5
6
7
8
9
|
[root@server02 ~]
# mkdir -p /glusterfs-xfs-mount
[root@server02 ~]
# mount /dev/sdb1 /glusterfs-xfs-mount/
[root@server02 ~]
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3
193G 7.3G 176G 4% /
tmpfs 932M 0 932M 0%
/dev/shm
/dev/sda1
190M 41M 139M 23%
/boot
/dev/sdb1
20G 44M 19G 1%
/glusterfs-xfs-mount
[root@server02 ~]
# echo '/dev/sdb1 /glusterfs-xfs-mount xfs defaults 0 0' >> /etc/fstab
|
6、配置yum源，并安装GlusterFS服务端软件
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
[root@server02 ~]
# cd /etc/yum.repos.d/
[root@server02 yum.repos.d]
# wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glust
erfs-epel.repo
--2016-04-18 14:32:22-- http:
//download
.gluster.org
/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel
.
repo
Resolving download.gluster.org... 23.253.208.221, 2001:4801:7824:104:be76:4eff:fe10:23d8
Connecting to download.gluster.org|23.253.208.221|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1049 (1.0K)
Saving to: “glusterfs-epel.repo”
100%[==============================================================>] 1,049 --.-K
/s
in
0s
2016-04-18 14:32:23 (36.4 MB
/s
) - “glusterfs-epel.repo” saved [1049
/1049
]
[root@server02 yum.repos.d]
# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noa
rch.rpm
Retrieving http:
//dl
.fedoraproject.org
/pub/epel/6/x86_64/epel-release-6-8
.noarch.rpm
warning:
/var/tmp/rpm-tmp
.gaJCKd: Header V3 RSA
/SHA256
Signature, key ID 0608b895: NOKEY
Preparing...
########################################### [100%]
1:epel-release
########################################### [100%]
[root@server02 yum.repos.d]
# yum -y install glusterfs-server
|
7、启动glusterd服务
|
1
2
3
4
|