实验拓扑：
一、安装配置Ceph服务
1.配置服务器时间同步
安装一台节点为主时间服务器，配置如下
其余服务器设置上游地址为主服务器
启动服务即可，chronyc sources查看
2.安装Ceph创建集群
在node1节点安装ceph-deploy
[root@ceph-node1 yum.repos.d]# yum -y install ceph-deploy
若无则创建/etc/ceph配置文件夹，且所有ceph-deploy命令的执行都要在该文件夹下
#进入/etc/ceph，创建ceph集群
[root@ceph-node1 ~]# mkdir /etc/ceph [root@ceph-node1 ~]# cd /etc/ceph/ [root@ceph-node1 ceph]# ceph-deploy new ceph-node1
集群创建完成后，工作目录下会创建集群配置文件和密钥文件
#在node1上执行install命令，可以直接在所有节点安装ceph软件包
[root@ceph-node1 ceph]# ceph-deploy install ceph-node1 ceph-node2 ceph-node3
#安装完成后，可以输入ceph -v查看版本检查是否安装成功
[root@ceph-node1 ~]# ceph -v ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43) [root@ceph-node2 ~]# ceph -v ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43) [root@ceph-node3 ~]# ceph -v ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
#在node1节点上创建第一个monitor
[root@ceph-node1 ceph]# ceph-deploy --overwrite-conf mon create-initial
#检查集群状态，可以看到此时处于不健康状态
[root@ceph-node1 ceph]# ceph -s cluster 37b6fbec-7562-404a-8a43-2e443f53937e health HEALTH_ERR 64 pgs stuck inactive 64 pgs stuck unclean no osds monmap e1: 1 mons at {ceph-node1=192.168.1.31:6789/0} election epoch 2, quorum 0 ceph-node1 osdmap e1: 0 osds: 0 up, 0 in pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects 0 kB used, 0 kB / 0 kB avail 64 creating
3.配置磁盘创建OSD
#disk list 列出节点上的所有可用磁盘，这里sdb为为服务器新加的50G磁盘
设置标签
[root@ceph-node1 ceph]# ceph-deploy disk list ceph-node1 [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf 。。。。。。 [ceph-node1][DEBUG ] /dev/sdb other, unknown
#对磁盘进行分区：parted磁盘分区，mklabel修改卷标，mkpart分区
[root@ceph-node1 ceph]# parted /dev/sdb GNU Parted 3.1 使用 /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel 新的磁盘标签类型？gpt 是/Yes/否/No? yes (parted) mkpart 分区名称？ []? 文件系统类型？ [ext2]? 起始点？0% 结束点？100% (parted) p #输出当前分区信息 Model: VMware, VMware Virtual S (scsi) Disk /dev/sdb: 53.7GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name 标志 1 1049kB 53.7GB 53.7GB (parted) q 信息: You may need to update /etc/fstab. [root@ceph-node1 ceph]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdb 8:16 0 50G 0 disk └─sdb1 8:17 0 50G 0 part
#对分区进行格式化，创建挂载目录进行挂载，并将文件夹权限配置为777
[root@ceph-node1 ceph]# mkfs.xfs /dev/sdb1 [root@ceph-node1 ceph]# mkdir /opt/osd1 [root@ceph-node1 ceph]# mount /dev/sdb1 /opt/osd1/ [root@ceph-node1 ceph]# chmod 777 /opt/osd1/
三个node节点都把磁盘进行相同操作
#在node1节点创建OSD节点
[root@ceph-node1 ceph]# ceph-deploy osd prepare ceph-node1:/opt/osd1 ceph-node2:/opt/osd2 ceph-node3:/opt/osd3
创建完成后，在三个节点中，修改osd下所有文件的权限为777
#确认权限修改777后，激活OSD节点
[root@ceph-node1 ceph]# ceph-deploy osd activate ceph-node1:/opt/osd1/ ceph-node2:/opt/osd2 ceph-node3:/opt/osd3
#再次查看集群状态，此时为健康模式
[root@ceph-node1 ceph]# ceph -s cluster 37b6fbec-7562-404a-8a43-2e443f53937e health HEALTH_OK monmap e1: 1 mons at {ceph-node1=192.168.1.31:6789/0} election epoch 2, quorum 0 ceph-node1 osdmap e13: 3 osds: 3 up, 3 in pgmap v19: 64 pgs, 1 pools, 0 bytes data, 0 objects 15460 MB used, 134 GB / 149 GB avail 64 active+clean
#最后开放权限给其他节点
[root@ceph-node1 ceph]# ceph-deploy admin ceph-node{1,2,3}
二、Ceph测试使用
1.安装ceph客户端
搭建一台Ceph-Client测试机：配置网络及yum源，配置主机名和hosts文件
[root@ceph-node1 ceph]# echo '192.168.1.34 ceph-client' >> /etc/hosts
#在node1节点上安装client上所需的ceph软件
[root@ceph-node1 ceph]# ceph-deploy install ceph-client
#在node1节点上开放权限给client
[root@ceph-node1 ceph]# ceph-deploy admin ceph-client
2.Ceph客户端使用
#rbd creata创建块设备镜像image，map映射image为块设备
[root@ceph-client ~]# rbd create foo --size 4096 -m ceph-node1 -k /etc/ceph/ceph.client.admin.keyring [root@ceph-client ~]# rbd map foo --name client.admin -m ceph-node1 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0
#rbd0块设备格式化并挂载，正常使用
[root@ceph-client ~]# mkfs.xfs /dev/rbd0 meta-data=/dev/rbd0 isize=256 agcount=9, agsize=130048 blks = sectsz=512 attr=2, projid32bit=1 = crc=0 finobt=0 data = bsize=4096 blocks=1048576, imaxpct=25 = sunit=1024 swidth=1024 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=0 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=8 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@ceph-client ~]# mkdir /mnt/test [root@ceph-client ~]# mount /dev/rbd0 /mnt/test [root@ceph-client ~]# df -h|grep rbd0 /dev/rbd0 4.0G 33M 4.0G 1% /mnt/test
3.调整块设备大小
#info foo可以查看当前块设备信息，此时容量为4G
[root@ceph-client ~]# rbd info foo rbd image 'foo': size 4096 MB in 1024 objects order 22 (4096 kB objects) block_name_prefix: rb.0.1026.238e1f29 format: 1 #使用resize命令将大小修改至10G [root@ceph-client ~]# rbd resize --size 10240 foo Resizing image: 100% complete...done. [root@ceph-client ~]# rbd info foo rbd image 'foo': size 10240 MB in 2560 objects order 22 (4096 kB objects) block_name_prefix: rb.0.1026.238e1f29 format: 1
特别注意的是，调整文件系统命令：
resize2fs命令 针对的是ext2、ext3、ext4文件系统
xfs_growfs命令 针对的是xfs文件系统
#执行调整文件系统大小，即可查看文件大小被扩容至10G
[root@ceph-client ~]# xfs_growfs /dev/rbd0 [root@ceph-client ~]# df -h|grep rbd0 /dev/rbd0 10G 33M 10G 1% /mnt/test
4.删除块设备
#1、取消块设备挂载 ->2、取消镜像与块设备的映射 -> 3、删除设备镜像
[root@ceph-client ~]# df -h|grep rbd0 /dev/rbd0 10G 33M 10G 1% /mnt/test [root@ceph-client ~]# umount /dev/rbd0 [root@ceph-client ~]# rbd unmap /dev/rbd/rbd/foo [root@ceph-client ~]# rbd rm foo Removing image: 100% complete...done.
三、Ceph命令
1.检查ceph安装状态
[root@ceph-node1 ceph]# ceph status cluster 37b6fbec-7562-404a-8a43-2e443f53937e health HEALTH_OK monmap e1: 1 mons at {ceph-node1=192.168.1.31:6789/0} election epoch 2, quorum 0 ceph-node1 osdmap e13: 3 osds: 3 up, 3 in pgmap v20: 64 pgs, 1 pools, 0 bytes data, 0 objects 15459 MB used, 134 GB / 149 GB avail 64 active+clean
2.检查monitor仲裁状态
[root@ceph-node1 ceph]# ceph quorum_status --format json-pretty { "election_epoch": 2, "quorum": [ 0 ], "quorum_names": [ "ceph-node1" ], "quorum_leader_name": "ceph-node1", "monmap": { "epoch": 1, "fsid": "37b6fbec-7562-404a-8a43-2e443f53937e", "modified": "0.000000", "created": "0.000000", "mons": [ { "rank": 0, "name": "ceph-node1", "addr": "192.168.1.31:6789\/0" } ] } }
3.导出monitor信息
[root@ceph-node1 ceph]# ceph mon dump dumped monmap epoch 1 epoch 1 fsid 37b6fbec-7562-404a-8a43-2e443f53937e last_changed 0.000000 created 0.000000 0: 192.168.1.31:6789/0 mon.ceph-node1
4.查看集群使用
[root@ceph-node1 ceph]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 149G 134G 15459M 10.07 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 46019M 0
5.检查monitor、OSD、PG配置组状态
[root@ceph-node1 ceph]# ceph mon stat e1: 1 mons at {ceph-node1=192.168.1.31:6789/0}, election epoch 2, quorum 0 ceph-node1 [root@ceph-node1 ceph]# ceph osd stat osdmap e13: 3 osds: 3 up, 3 in [root@ceph-node1 ceph]# ceph pg stat v20: 64 pgs: 64 active+clean; 0 bytes data, 15459 MB used, 134 GB / 149 GB avail
6.列出PG
[root@ceph-node1 ceph]# ceph pg dump
7.列出ceph存储池
[root@ceph-node1 ceph]# ceph osd lspools 0 rbd,
8.检查OSD的crush
[root@ceph-node1 ceph]# ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 0.14996 root default -2 0.04999 host ceph-node1 0 0.04999 osd.0 up 1.00000 1.00000 -3 0.04999 host ceph-node2 1 0.04999 osd.1 up 1.00000 1.00000 -4 0.04999 host ceph-node3 2 0.04999 osd.2 up 1.00000 1.00000
9.列出集群的认证密钥
[root@ceph-node1 ceph]# ceph auth list installed auth entries: osd.0 key: AQBFsx1f/+qKFRAAyLOYuF7ep/FXLMI5yca/zg== caps: [mon] allow profile osd caps: [osd] allow * osd.1 key: AQBRsx1fXCGlHBAAViSH6eCjRI7NbrLFWGajrQ== caps: [mon] allow profile osd caps: [osd] allow * osd.2 key: AQBksx1fgjK9NRAAe827GWfDqfMtwxRqJ3D0sA== caps: [mon] allow profile osd caps: [osd] allow * client.admin key: AQAsrh1fAngFJhAAKPsed40MDoLgbvk7L4kuJg== caps: [mds] allow caps: [mon] allow * caps: [osd] allow * client.bootstrap-mds key: AQAtrh1fcaafBxAA3EOMSIIlggVmUwPs6lIFzw== caps: [mon] allow profile bootstrap-mds client.bootstrap-osd key: AQAsrh1fYXJsMRAAQR1fAeuSAfJ+ze4YHei7Hw== caps: [mon] allow profile bootstrap-osd client.bootstrap-rgw key: AQAsrh1f6O+BORAAiWjtCwFnVgkjaVfFQXtTKQ== caps: [mon] allow profile bootstrap-rgw
