一.rbd块设备的增删改查

推荐阅读:
    https://docs.ceph.com/en/nautilus/rbd/rados-rbd-cmds/

1.创建存储池

[root@ceph141 ~]# ceph osd pool create yinzhengjie-rbd 16 16
pool 'yinzhengjie-rbd' created
[root@ceph141 ~]#

2.使用rbd工具初始化池以供RBD块设备使用

    1.对存储池启用rbd功能.
[root@ceph141 ~]# ceph osd pool application get yinzhengjie-rbd
{}
[root@ceph141 ~]#  
[root@ceph141 ~]# ceph osd pool application enable yinzhengjie-rbd rbd
enabled application 'rbd' on pool 'yinzhengjie-rbd'
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd pool application get yinzhengjie-rbd
{
    "rbd": {}
}
[root@ceph141 ~]# 


    2.对存储池进行环境初始化
[root@ceph141 ~]# rbd pool init yinzhengjie-rbd
[root@ceph141 ~]#
[root@ceph141 ~]# rbd pool stats yinzhengjie-rbd
Total Images: 0
Total Snapshots: 0
Provisioned Size: 0 B
[root@ceph141 ~]# 

温馨提示:
    - 1.如果rbd存储池不启用功能，则使用"ceph -s"命令查看时就会提示报错"1 pool(s) do not have an application enabled"信息；
    - 2.可以使用"osd pool application enable <pool> <app>"来启用存储池的app功能，其中app的有效值为: "cephfs","rbd","rgw"；

3.创建一个2G的rbd设备

[root@ceph141 ~]# rbd create wordpress --size 2048  -p yinzhengjie-rbd

4.查看rbd块设备

[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd
wordpress
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME      SIZE  PARENT FMT PROT LOCK 
wordpress 2 GiB          2           
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l --format json
[{"image":"wordpress","size":2147483648,"format":2}]
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l --format json --pretty-format
[
    {
        "image": "wordpress",
        "size": 2147483648,
        "format": 2
    }
]
[root@ceph141 ~]#

5. 查看镜像的详细信息

[root@ceph141 ~]# rbd info  -p yinzhengjie-rbd wordpress
rbd image 'wordpress':
    size 2 GiB in 512 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: 12dc1b0f25c9
    block_name_prefix: rbd_data.12dc1b0f25c9
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    op_features: 
    flags: 
    create_timestamp: Thu Feb  1 11:42:11 2024
    access_timestamp: Thu Feb  1 11:42:11 2024
    modify_timestamp: Thu Feb  1 11:42:11 2024
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd info  yinzhengjie-rbd/wordpress
rbd image 'wordpress':
    size 2 GiB in 512 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: 12dc1b0f25c9
    block_name_prefix: rbd_data.12dc1b0f25c9
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    op_features: 
    flags: 
    create_timestamp: Thu Feb  1 11:42:11 2024
    access_timestamp: Thu Feb  1 11:42:11 2024
    modify_timestamp: Thu Feb  1 11:42:11 2024
[root@ceph141 ~]# 


常用的镜像features属性说明:
      - layering:
          分层克隆机制，磁盘的数据分层获取克隆机制。
     - striping:
          是否支持数据对象间的数据条带化。
     - exclusive-lock:
         排它锁的机制，磁盘应用于多路写机制场景，限制同时仅能有有一个客户端访问当前image。
     - object-map:
         对象位图机制，主要用于加速导入，导出及已用容量统计等操作，依赖于"exclusive-lock"特性。
     - fast-diff:
         快照定制机制，快速对比数据差异，便于做快照管理，依赖于object-map特性。
     - deep-flatten:
         数据处理机制，解除父子image及快照的依赖关系。
     - journaling:
         磁盘日志机制，将image的所有修改操作进行日志话，便于异地备份，依赖于"exclusive-lock"特性。
     - data-pool:
         是否支持将image的数据对象存储于纠删码存储池，主要用于将image的元数据于数据放置于不同的存储池。

6.修改rbd

    1 修改rbd的名称
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME      SIZE  PARENT FMT PROT LOCK 
wordpress 2 GiB          2           
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd rename  wordpress wp -p yinzhengjie-rbd
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME SIZE  PARENT FMT PROT LOCK 
wp   2 GiB          2           
[root@ceph141 ~]# 


    2 修改rbd的大小
[root@ceph141 ~]# rbd ls --pool yinzhengjie-rbd -l
NAME SIZE  PARENT FMT PROT LOCK 
wp   2 GiB          2           
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd resize  -s 4G yinzhengjie-rbd/wp
Resizing image: 100% complete...done.
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME SIZE  PARENT FMT PROT LOCK 
wp   4 GiB          2           
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd resize  --size 10G --pool yinzhengjie-rbd --image wp
Resizing image: 100% complete...done.
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME SIZE   PARENT FMT PROT LOCK 
wp   10 GiB          2           
[root@ceph141 ~]#

7.删除块设备

    基于回收站临时删除块设备：（回收站，推荐，【但并不释放空间】）
推荐阅读:
      https://www.cnblogs.com/yinzhengjie/p/18383834

    永久删除块设备
[root@ceph151 ~]# rbd ls --pool yinzhengjie-rbd -l
NAME  SIZE    PARENT  FMT  PROT  LOCK
wp    10 GiB            2            
[root@ceph151 ~]# 
[root@ceph151 ~]# rbd rm yinzhengjie-rbd/wp
Removing image: 100% complete...done.
[root@ceph151 ~]# 
[root@ceph151 ~]# rbd ls --pool yinzhengjie-rbd -l
[root@ceph151 ~]#

二.CentOS环境ceph客户端使用rbd实战

1.安装ceph的客户端模块

[root@harbor250 ~]# cat /etc/yum.repos.d/ceph.repo 
[Ceph]
name=Ceph packages for 
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/$basearch
gpgcheck=0
[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch
gpgcheck=0
[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/SRPMS
gpgcheck=0
[root@harbor250 ~]# 
[root@harbor250 ~]# yum -y install ceph-common

2.在RBD存储池创建块设备

    1 创建镜像时可以指定镜像的特性
[root@ceph143 ~]# rbd create -s 2G yinzhengjie-rbd/wordpress
[root@ceph143 ~]# 
[root@ceph143 ~]# rbd create -s 5G -p yinzhengjie-rbd --image mysqld --image-feature layering,exclusive-lock
[root@ceph143 ~]# 
[root@ceph143 ~]# rbd ls -l -p yinzhengjie-rbd 
NAME      SIZE  PARENT FMT PROT LOCK 
mysqld    5 GiB          2           
wordpress 2 GiB          2           
[root@ceph143 ~]# 


    2 查看镜像的详细信息
[root@ceph142 ~]# rbd info -p yinzhengjie-rbd wordpress | grep '^[[:space:]]features'
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
[root@ceph142 ~]# 
[root@ceph142 ~]# rbd info -p yinzhengjie-rbd mysqld | grep "\sfeatures"
    features: layering, exclusive-lock
[root@ceph142 ~]#

3.修改RBD的镜像特性，由于centos 7 Linux内核版本过低(3.10.0-1127.el7.x86_64)，不支持更多的特性

[root@ceph141 ~]# rbd info -p yinzhengjie-rbd wordpress | grep '^[[:space:]]features'
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd feature disable yinzhengjie-rbd/wordpress object-map fast-diff deep-flatten
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd info -p yinzhengjie-rbd wordpress | grep '\sfeatures'
    features: layering, exclusive-lock
[root@ceph141 ~]#

4.客户端映射镜像(块设备)

    1 拷贝ceph的配置文件，主要用于识别ceph集群地址
[root@harbor250 ~]# scp ceph141:/etc/ceph/ceph.conf /etc/ceph/
ceph.conf                                                                                            100%  264   113.6KB/s   00:00    
[root@harbor250 ~]# 
[root@harbor250 ~]# grep mon_host /etc/ceph/ceph.conf 
mon_host = 10.0.0.141,10.0.0.142,10.0.0.143
[root@harbor250 ~]# 

    2 拷贝ceph的认证文件，主要用于ceph集群认证
[root@harbor250 ~]# scp ceph141:/etc/ceph/ceph.client.admin.keyring /etc/ceph/
ceph.client.admin.keyring                                                                            100%  151   111.1KB/s   00:00    
[root@harbor250 ~]# 
[root@harbor250 ~]# grep key /etc/ceph/ceph.client.admin.keyring 
    key = AQDjFrplyvFCDhAApJg111YMIGQ6/F/x/Y+qpQ==
[root@harbor250 ~]# 


    3 添加映射
[root@harbor250 ~]# ll /dev/rbd*
ls: cannot access /dev/rbd*: No such file or directory
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd -p yinzhengjie-rbd map mysqld
/dev/rbd0
[root@harbor250 ~]#  
[root@harbor250 ~]# ll /dev/rbd*
brw-rw---- 1 root disk 252, 0 Feb  1 12:08 /dev/rbd0

/dev/rbd:
total 0
drwxr-xr-x 2 root root 60 Feb  1 12:08 yinzhengjie-rbd
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd -p yinzhengjie-rbd map wordpress
/dev/rbd1
[root@harbor250 ~]# 
[root@harbor250 ~]# ll /dev/rbd*
brw-rw---- 1 root disk 252,  0 Feb  1 12:08 /dev/rbd0
brw-rw---- 1 root disk 252, 16 Feb  1 12:14 /dev/rbd1

/dev/rbd:
total 0
drwxr-xr-x 2 root root 80 Feb  1 12:14 yinzhengjie-rbd
[root@harbor250 ~]#

5.对块设备进行格式化

[root@harbor250 ~]# mkfs.xfs /dev/rbd0 
meta-data=/dev/rbd0              isize=512    agcount=8, agsize=163840 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=1310720, imaxpct=25
         =                       sunit=1024   swidth=1024 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@harbor250 ~]# 
[root@harbor250 ~]# mkfs.ext4 /dev/rbd1
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done                            
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1024 blocks, Stripe width=1024 blocks
131072 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

[root@harbor250 ~]#

6.创建挂载点

[root@harbor250 ~]# mkdir -pv /yinzhengjie/data/{wordpress,mysql}
mkdir: created directory ‘/yinzhengjie/data’
mkdir: created directory ‘/yinzhengjie/data/wordpress’
mkdir: created directory ‘/yinzhengjie/data/mysql’
[root@harbor250 ~]#

7.尝试手动挂载

[root@harbor250 ~]# mount /dev/rbd0 /yinzhengjie/data/mysql
[root@harbor250 ~]# 
[root@harbor250 ~]# mount /dev/rbd1 /yinzhengjie/data/wordpress
[root@harbor250 ~]# 
[root@harbor250 ~]# df -h | grep rbd
/dev/rbd0                5.0G   33M  5.0G   1% /yinzhengjie/data/mysql
/dev/rbd1                2.0G  6.0M  1.8G   1% /yinzhengjie/data/wordpress
[root@harbor250 ~]#

8.配置ceph开机自启动，(做完实验后可以注释掉，以免当你删除存储池后，无法挂载块设备的问题。)

[root@harbor250 ~]# vim /etc/rc.d/rc.local 

...
rbd -p yinzhengjie-rbd map mysqld
rbd -p yinzhengjie-rbd map wordpress
mount /dev/rbd0 /yinzhengjie/data/mysql
mount /dev/rbd1 /yinzhengjie/data/wordpress
[root@harbor250 ~]# 
[root@harbor250 ~]# ll  /etc/rc.d/rc.local 
-rw-r--r-- 1 root root 620 Feb  1 12:21 /etc/rc.d/rc.local
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# chmod +x /etc/rc.d/rc.local 
[root@harbor250 ~]# 
[root@harbor250 ~]# ll  /etc/rc.d/rc.local 
-rwxr-xr-x 1 root root 620 Feb  1 12:21 /etc/rc.d/rc.local
[root@harbor250 ~]# 
[root@harbor250 ~]# df -h | grep rbd
/dev/rbd0                5.0G   33M  5.0G   1% /yinzhengjie/data/mysql
/dev/rbd1                2.0G  6.0M  1.8G   1% /yinzhengjie/data/wordpress
[root@harbor250 ~]# 
[root@harbor250 ~]# umount /yinzhengjie/data/mysql 
[root@harbor250 ~]# 
[root@harbor250 ~]# df -h | grep rbd
/dev/rbd1                2.0G  6.0M  1.8G   1% /yinzhengjie/data/wordpress
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# umount /dev/rbd1 
[root@harbor250 ~]# 
[root@harbor250 ~]# df -h | grep rbd
[root@harbor250 ~]# 
[root@harbor250 ~]#  reboot 
[root@harbor250 ~]#
[root@harbor250 ~]# df -h | grep yinzhengjie
/dev/rbd0                5.0G   33M  5.0G   1% /yinzhengjie/data/mysql
/dev/rbd1                2.0G  6.0M  1.8G   1% /yinzhengjie/data/wordpress
[root@harbor250 ~]#

9.测试RBD块设备的可用性

[root@harbor250 ~]# cp /etc/os-release /yinzhengjie/data/mysql/
[root@harbor250 ~]# 
[root@harbor250 ~]# cp /etc/hosts /yinzhengjie/data/mysql/
[root@harbor250 ~]# 
[root@harbor250 ~]# cp /etc/fstab /yinzhengjie/data/wordpress/
[root@harbor250 ~]# 
[root@harbor250 ~]# ll /yinzhengjie/data/*
/yinzhengjie/data/mysql:
total 8
-rw-r--r-- 1 root root 288 Feb  1 12:24 hosts
-rw-r--r-- 1 root root 393 Feb  1 12:24 os-release

/yinzhengjie/data/wordpress:
total 20
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
[root@harbor250 ~]#

10.卸载设备

[root@harbor250 ~]# umount /yinzhengjie/data/mysql 
[root@harbor250 ~]# 
[root@harbor250 ~]# umount /yinzhengjie/data/wordpress 
[root@harbor250 ~]# 
[root@harbor250 ~]# df | grep  yinzhengjie
[root@harbor250 ~]#

11.其他客户端重新挂载设备观察数据是否丢失

[root@k8s-worker05 ~]# cat /etc/yum.repos.d/ceph.repo 
[Ceph]
name=Ceph packages for 
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/$basearch
gpgcheck=0
[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch
gpgcheck=0
[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/SRPMS
gpgcheck=0

[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# yum -y install ceph-common
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# scp 10.0.0.141:/etc/ceph/ceph.client.admin.keyring /etc/ceph/
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# scp 10.0.0.141:/etc/ceph/ceph.conf /etc/ceph/
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# rbd  map yinzhengjie-rbd/mysqld
/dev/rbd0
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# mount /dev/rbd0 /mnt/
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# ll /mnt/
total 8
-rw-r--r-- 1 root root 288 Feb  1 12:24 hosts
-rw-r--r-- 1 root root 393 Feb  1 12:24 os-release
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# mkdir /opt/yinzhengjie
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 0
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# rbd  map yinzhengjie-rbd/wordpress
/dev/rbd1
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# mount /dev/rbd1 /opt/yinzhengjie/
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 20
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
[root@k8s-worker05 ~]#

12.两个节点不得使用同一个块设备，否则可能会出现数据不一致的问题！各节点应该单独使用块设备。

    1.客户端1操作如下:
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 20
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# cp /etc/sysconfig/network-scripts/ifcfg-ens33 /opt/yinzhengjie/
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# cp /etc/resolv.conf /opt/yinzhengjie
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 28
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
-rw-r--r-- 1 root root   163 Feb  1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
-rw-r--r-- 1 root root    72 Feb  1 12:34 resolv.conf
[root@k8s-worker05 ~]# 


    2.客户端2操作如下:
[root@harbor250 ~]# mount /dev/rbd0 /yinzhengjie/data/mysql/
[root@harbor250 ~]# 
[root@harbor250 ~]# ll /yinzhengjie/data/mysql/
total 8
-rw-r--r-- 1 root root 288 Feb  1 12:24 hosts
-rw-r--r-- 1 root root 393 Feb  1 12:24 os-release
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# mount /dev/rbd1 /yinzhengjie/data/wordpress/
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# ll /yinzhengjie/data/wordpress/
total 28
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
-rw-r--r-- 1 root root   163 Feb  1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
-rw-r--r-- 1 root root    72 Feb  1 12:34 resolv.conf
[root@harbor250 ~]# 

    3.客户端1继续操作如下:
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 28
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
-rw-r--r-- 1 root root   163 Feb  1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
-rw-r--r-- 1 root root    72 Feb  1 12:34 resolv.conf
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# cp /etc/ceph/ceph.conf /opt/yinzhengjie
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 32
-rw-r--r-- 1 root root   264 Feb  1 12:36 ceph.conf
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
-rw-r--r-- 1 root root   163 Feb  1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
-rw-r--r-- 1 root root    72 Feb  1 12:34 resolv.conf
[root@k8s-worker05 ~]# 


    4.客户端2再次验证，不难发现，两个节点数据已经开始不一致啦！为了避免这种情况，不建议两个节点使用同一个块设备。
[root@harbor250 ~]# ll /yinzhengjie/data -R
/yinzhengjie/data:
total 4
drwxr-xr-x 2 root root   37 Feb  1 12:24 mysql
drwxr-xr-x 3 root root 4096 Feb  1 12:34 wordpress

/yinzhengjie/data/mysql:
total 8
-rw-r--r-- 1 root root 288 Feb  1 12:24 hosts
-rw-r--r-- 1 root root 393 Feb  1 12:24 os-release

/yinzhengjie/data/wordpress:
total 28
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
-rw-r--r-- 1 root root   163 Feb  1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
-rw-r--r-- 1 root root    72 Feb  1 12:34 resolv.conf

/yinzhengjie/data/wordpress/lost+found:
total 0
[root@harbor250 ~]#

13.温馨提示

[root@k8s-worker05 ~]# cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# uname -r
4.19.12-1.el7.elrepo.x86_64
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# rbd map yinzhengjie-rbd/k8s  # 尽管将内核升级为4系列的，发现依旧不好使，建议将内核升级为5系列的。
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable yinzhengjie-rbd/k8s object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
[root@k8s-worker05 ~]#

三. Ubuntu环境ceph客户端使用rbd实战

1.Ubuntu安装ceph环境

[root@prometheus-server32 ~]# wget -q -O- 'https://download.ceph.com/keys/release.asc' |  apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# echo deb https://download.ceph.com/debian-reef/ $(lsb_release -sc) main | tee /etc/apt/sources.list.d/ceph.list
deb https://download.ceph.com/debian-reef/ jammy main
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# apt update
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# apt -y install ceph-common
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# ceph --version
ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
[root@prometheus-server32 ~]# 


温馨提示:
    但是官方对于Ceph的Nautilus(V14.2.22)版本而言，最高支持的Ubuntu系统是: Ubuntu 20.04 LTS Focal ，最高支持的CentOS 7。
    由于我的系统是"Ubuntu 22.04 LTS Fammy"，不支持Nautilus(V14.2.22)版本安装。 
    因此我安装的Ceph reef(V18.2.1)版本，如果想要安装和我们自定义的Nautilus(V14.2.22)版本。需要降低Ubuntu系统的版本号并做如下调整。

[root@prometheus-server32 ~]# wget -q -O- 'https://download.ceph.com/keys/release.asc' |  apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# echo deb https://download.ceph.com/debian-nautilus/ $(lsb_release -sc) main | tee /etc/apt/sources.list.d/ceph.list
deb https://download.ceph.com/debian-nautilus/ jammy main
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# cat /etc/apt/sources.list.d/ceph.list 
deb https://download.ceph.com/debian-nautilus/ jammy main
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# apt update


    如果网速慢的同学，可以将"download.ceph.com"更换为"mirrors.aliyun.com"以提高速度。



参考链接:
    https://docs.ceph.com/en/latest/install/get-packages/#apt
    https://docs.ceph.com/en/latest/install/get-packages/#debian-packages

2.Ubuntu拷贝ceph文件及证书

[root@prometheus-server32 ~]# scp 10.0.0.141:/etc/ceph/ceph.c* /etc/ceph/
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# ceph -s
  cluster:
    id:     5821e29c-326d-434d-a5b6-c492527eeaad
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 21h)
    mgr: ceph141(active, since 21h), standbys: ceph143, ceph142
    osd: 7 osds: 7 up (since 21h), 7 in (since 21h)

  data:
    pools:   2 pools, 32 pgs
    objects: 43 objects, 86 MiB
    usage:   7.7 GiB used, 1.9 TiB / 2.0 TiB avail
    pgs:     32 active+clean

[root@prometheus-server32 ~]#

3.创建镜像设备

[root@prometheus-server32 ~]# rbd create -s 5G yinzhengjie-rbd/k8s 
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# rbd ls -p yinzhengjie-rbd
k8s
mysqld
wordpress
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# rbd info yinzhengjie-rbd/k8s 
rbd image 'k8s':
    size 5 GiB in 1280 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: 13fa4379e5c8
    block_name_prefix: rbd_data.13fa4379e5c8
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    op_features: 
    flags: 
    create_timestamp: Thu Feb  1 07:21:28 2024
    access_timestamp: Thu Feb  1 07:21:28 2024
    modify_timestamp: Thu Feb  1 07:21:28 2024
[root@prometheus-server32 ~]#

4.Ubuntu系统添加映射

[root@prometheus-server32 ~]# uname -r
5.15.0-92-generic
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# rbd info yinzhengjie-rbd/k8s | grep '\sfeatures'
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# rbd showmapped
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# rbd  map yinzhengjie-rbd/k8s
/dev/rbd0
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# rbd showmapped    # 查看本地的挂载信息。
id  pool             namespace  image  snap  device   
0   yinzhengjie-rbd             k8s    -     /dev/rbd0
[root@prometheus-server32 ~]#

5.对设备进行格式化并挂载

[root@prometheus-server32 ~]# mkfs.xfs  /dev/rbd0 
meta-data=/dev/rbd0              isize=512    agcount=8, agsize=163840 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=1310720, imaxpct=25
         =                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...Done.
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# mount /dev/rbd0 /mnt/
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# df -h | grep mnt
/dev/rbd0                          5.0G   69M  5.0G   2% /mnt
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# cp /etc/os-release /etc/hosts /mnt/
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# ll /mnt/
total 12
drwxr-xr-x  2 root root   37 Feb  1 07:25 ./
drwxr-xr-x 20 root root 4096 Jan 25 02:55 ../
-rw-r--r--  1 root root  226 Feb  1 07:25 hosts
-rw-r--r--  1 root root  386 Feb  1 07:25 os-release
[root@prometheus-server32 ~]#

6.卸载设备

[root@prometheus-server32 ~]# umount /mnt

7.centos系统尝试挂载设备并验证数据

[root@harbor250 ~]# uname -r
3.10.0-1127.el7.x86_64
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd  map yinzhengjie-rbd/k8s  # 发现挂载会失败，报错所不支持特性
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable yinzhengjie-rbd/k8s object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd info yinzhengjie-rbd/k8s | grep '\sfeatures'
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd feature disable yinzhengjie-rbd/k8s object-map fast-diff deep-flatten  # 根据上面的报错禁用特性
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd info yinzhengjie-rbd/k8s | grep '\sfeatures'
    features: layering, exclusive-lock
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd  map yinzhengjie-rbd/k8s  # 禁用特性后，发现可以映射了，映射的设备类型为"/dev/rbd2"
/dev/rbd2
[root@harbor250 ~]# 
[root@harbor250 ~]# mount /dev/rbd2 /mnt/  # 很明显，挂载失败了，说是不支持块设备。
mount: wrong fs type, bad option, bad superblock on /dev/rbd2,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
[root@harbor250 ~]# 
[root@harbor250 ~]# mkfs.xfs /dev/rbd2   # 此处我重新格式化，发现说已经存在了xfs系统，因此需要强行格式化
mkfs.xfs: /dev/rbd2 appears to contain an existing filesystem (xfs).
mkfs.xfs: Use the -f option to force overwrite.
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# mkfs.xfs /dev/rbd2  -f  # 强行格式化后，数据会丢失
meta-data=/dev/rbd2              isize=512    agcount=8, agsize=163840 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=1310720, imaxpct=25
         =                       sunit=1024   swidth=1024 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@harbor250 ~]# 
[root@harbor250 ~]# mount /dev/rbd2 /mnt/  # 格式化后再次挂载，发现数据丢失了。
[root@harbor250 ~]# 
[root@harbor250 ~]# ll /mnt/
total 0
[root@harbor250 ~]# 
[root@harbor250 ~]# cp /etc/sysconfig/network-scripts/ifcfg-ens33 /mnt/   # 添加数据
[root@harbor250 ~]# 
[root@harbor250 ~]# ll /mnt/
total 4
-rw-r--r-- 1 root root 147 Feb  1 15:32 ifcfg-ens33
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# cat /mnt/ifcfg-ens33 
TYPE="Ethernet"
BOOTPROTO="static"
NAME="ens33"
DEVICE="ens33"
ONBOOT="yes"
IPADDR="10.0.0.250"
PREFIX="24"
GATEWAY="10.0.0.254"
DNS1="10.0.0.254"
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# umount /mnt 
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd  unmap yinzhengjie-rbd/k8s  # 取消映射
[root@harbor250 ~]#

ceph的rbd块设备的使用指南

一.rbd块设备的增删改查

1.创建存储池

2.使用rbd工具初始化池以供RBD块设备使用

3.创建一个2G的rbd设备

4.查看rbd块设备

5. 查看镜像的详细信息

6.修改rbd

7.删除块设备

二.CentOS环境ceph客户端使用rbd实战

1.安装ceph的客户端模块

2.在RBD存储池创建块设备

3.修改RBD的镜像特性，由于centos 7 Linux内核版本过低(3.10.0-1127.el7.x86_64)，不支持更多的特性

4.客户端映射镜像(块设备)

5.对块设备进行格式化

6.创建挂载点

7.尝试手动挂载

8.配置ceph开机自启动，(做完实验后可以注释掉，以免当你删除存储池后，无法挂载块设备的问题。)

9.测试RBD块设备的可用性

10.卸载设备

11.其他客户端重新挂载设备观察数据是否丢失

12.两个节点不得使用同一个块设备，否则可能会出现数据不一致的问题！各节点应该单独使用块设备。

13.温馨提示

三. Ubuntu环境ceph客户端使用rbd实战

1.Ubuntu安装ceph环境

2.Ubuntu拷贝ceph文件及证书

3.创建镜像设备

4.Ubuntu系统添加映射

5.对设备进行格式化并挂载

6.卸载设备

7.centos系统尝试挂载设备并验证数据

热门文章

最新文章

相关课程

相关电子书

相关实验场景

热门

活动广场

任务中心

开发者评测

高校计划

乘风者计划

训练营

阿里云MVP

话题

直播

下载

镜像站

技术资料

插件

ceph的rbd块设备的使用指南

一.rbd块设备的增删改查

1.创建存储池

2.使用rbd工具初始化池以供RBD块设备使用

3.创建一个2G的rbd设备

4.查看rbd块设备

5. 查看镜像的详细信息

6.修改rbd

7.删除块设备

二.CentOS环境ceph客户端使用rbd实战

1.安装ceph的客户端模块

2.在RBD存储池创建块设备

3.修改RBD的镜像特性，由于centos 7 Linux内核版本过低(3.10.0-1127.el7.x86_64)，不支持更多的特性

4.客户端映射镜像(块设备)

5.对块设备进行格式化

6.创建挂载点

7.尝试手动挂载

8.配置ceph开机自启动，(做完实验后可以注释掉，以免当你删除存储池后，无法挂载块设备的问题。)

9.测试RBD块设备的可用性

10.卸载设备

11.其他客户端重新挂载设备观察数据是否丢失

12.两个节点不得使用同一个块设备，否则可能会出现数据不一致的问题！各节点应该单独使用块设备。

13.温馨提示

三. Ubuntu环境ceph客户端使用rbd实战

1.Ubuntu安装ceph环境

2.Ubuntu拷贝ceph文件及证书

3.创建镜像设备

4.Ubuntu系统添加映射

5.对设备进行格式化并挂载

6.卸载设备

7.centos系统尝试挂载设备并验证数据

热门文章

最新文章

相关课程

相关电子书

相关实验场景