AI 助理
备案控制台
开发者社区 开发与运维 文章 正文

ceph的rbd块设备的使用指南

2024-09-03 77
版权
版权声明：
本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《 阿里云开发者社区用户服务协议》和 《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写 侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
本文涉及的产品
RDS MySQL Serverless 基础系列，0.5-2RCU 50GB
推荐场景：
学生管理系统数据库设计 搭建个人博客
Redis 开源版，标准版 2GB
推荐场景：
搭建游戏排行榜
云数据库 Tair（兼容Redis），内存型 2GB
推荐场景：
通过缓存加速数据库访问
简介： 这篇文章介绍了如何在CentOS和Ubuntu环境下使用Ceph的RBD（Rados Block Device）进行块设备的创建、修改、删除和挂载操作。

一.rbd块设备的增删改查

推荐阅读:
    https://docs.ceph.com/en/nautilus/rbd/rados-rbd-cmds/

1.创建存储池

[root@ceph141 ~]# ceph osd pool create yinzhengjie-rbd 16 16
pool 'yinzhengjie-rbd' created
[root@ceph141 ~]#

2.使用rbd工具初始化池以供RBD块设备使用

    1.对存储池启用rbd功能.
[root@ceph141 ~]# ceph osd pool application get yinzhengjie-rbd
{}
[root@ceph141 ~]#  
[root@ceph141 ~]# ceph osd pool application enable yinzhengjie-rbd rbd
enabled application 'rbd' on pool 'yinzhengjie-rbd'
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd pool application get yinzhengjie-rbd
{
    "rbd": {}
}
[root@ceph141 ~]# 


    2.对存储池进行环境初始化
[root@ceph141 ~]# rbd pool init yinzhengjie-rbd
[root@ceph141 ~]#
[root@ceph141 ~]# rbd pool stats yinzhengjie-rbd
Total Images: 0
Total Snapshots: 0
Provisioned Size: 0 B
[root@ceph141 ~]# 

温馨提示:
    - 1.如果rbd存储池不启用功能，则使用"ceph -s"命令查看时就会提示报错"1 pool(s) do not have an application enabled"信息；
    - 2.可以使用"osd pool application enable <pool> <app>"来启用存储池的app功能，其中app的有效值为: "cephfs","rbd","rgw"；

3.创建一个2G的rbd设备

[root@ceph141 ~]# rbd create wordpress --size 2048  -p yinzhengjie-rbd

4.查看rbd块设备

[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd
wordpress
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME      SIZE  PARENT FMT PROT LOCK 
wordpress 2 GiB          2           
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l --format json
[{"image":"wordpress","size":2147483648,"format":2}]
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l --format json --pretty-format
[
    {
        "image": "wordpress",
        "size": 2147483648,
        "format": 2
    }
]
[root@ceph141 ~]#

5. 查看镜像的详细信息

[root@ceph141 ~]# rbd info  -p yinzhengjie-rbd wordpress
rbd image 'wordpress':
    size 2 GiB in 512 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: 12dc1b0f25c9
    block_name_prefix: rbd_data.12dc1b0f25c9
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    op_features: 
    flags: 
    create_timestamp: Thu Feb  1 11:42:11 2024
    access_timestamp: Thu Feb  1 11:42:11 2024
    modify_timestamp: Thu Feb  1 11:42:11 2024
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd info  yinzhengjie-rbd/wordpress
rbd image 'wordpress':
    size 2 GiB in 512 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: 12dc1b0f25c9
    block_name_prefix: rbd_data.12dc1b0f25c9
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    op_features: 
    flags: 
    create_timestamp: Thu Feb  1 11:42:11 2024
    access_timestamp: Thu Feb  1 11:42:11 2024
    modify_timestamp: Thu Feb  1 11:42:11 2024
[root@ceph141 ~]# 


常用的镜像features属性说明:
      - layering:
          分层克隆机制，磁盘的数据分层获取克隆机制。
     - striping:
          是否支持数据对象间的数据条带化。
     - exclusive-lock:
         排它锁的机制，磁盘应用于多路写机制场景，限制同时仅能有有一个客户端访问当前image。
     - object-map:
         对象位图机制，主要用于加速导入，导出及已用容量统计等操作，依赖于"exclusive-lock"特性。
     - fast-diff:
         快照定制机制，快速对比数据差异，便于做快照管理，依赖于object-map特性。
     - deep-flatten:
         数据处理机制，解除父子image及快照的依赖关系。
     - journaling:
         磁盘日志机制，将image的所有修改操作进行日志话，便于异地备份，依赖于"exclusive-lock"特性。
     - data-pool:
         是否支持将image的数据对象存储于纠删码存储池，主要用于将image的元数据于数据放置于不同的存储池。

6.修改rbd

    1 修改rbd的名称
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME      SIZE  PARENT FMT PROT LOCK 
wordpress 2 GiB          2           
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd rename  wordpress wp -p yinzhengjie-rbd
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME SIZE  PARENT FMT PROT LOCK 
wp   2 GiB          2           
[root@ceph141 ~]# 


    2 修改rbd的大小
[root@ceph141 ~]# rbd ls --pool yinzhengjie-rbd -l
NAME SIZE  PARENT FMT PROT LOCK 
wp   2 GiB          2           
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd resize  -s 4G yinzhengjie-rbd/wp
Resizing image: 100% complete...done.
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME SIZE  PARENT FMT PROT LOCK 
wp   4 GiB          2           
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd resize  --size 10G --pool yinzhengjie-rbd --image wp
Resizing image: 100% complete...done.
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME SIZE   PARENT FMT PROT LOCK 
wp   10 GiB          2           
[root@ceph141 ~]#

7.删除块设备

    基于回收站临时删除块设备：（回收站，推荐，【但并不释放空间】）
推荐阅读:
      https://www.cnblogs.com/yinzhengjie/p/18383834

    永久删除块设备
[root@ceph151 ~]# rbd ls --pool yinzhengjie-rbd -l
NAME  SIZE    PARENT  FMT  PROT  LOCK
wp    10 GiB            2            
[root@ceph151 ~]# 
[root@ceph151 ~]# rbd rm yinzhengjie-rbd/wp
Removing image: 100% complete...done.
[root@ceph151 ~]# 
[root@ceph151 ~]# rbd ls --pool yinzhengjie-rbd -l
[root@ceph151 ~]#

二.CentOS环境ceph客户端使用rbd实战

1.安装ceph的客户端模块

[root@harbor250 ~]# cat /etc/yum.repos.d/ceph.repo 
[Ceph]
name=Ceph packages for 
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/$basearch
gpgcheck=0
[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch
gpgcheck=0
[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/SRPMS
gpgcheck=0
[root@harbor250 ~]# 
[root@harbor250 ~]# yum -y install ceph-common

2.在RBD存储池创建块设备

    1 创建镜像时可以指定镜像的特性
[root@ceph143 ~]# rbd create -s 2G yinzhengjie-rbd/wordpress
[root@ceph143 ~]# 
[root@ceph143 ~]# rbd create -s 5G -p yinzhengjie-rbd --image mysqld --image-feature layering,exclusive-lock
[root@ceph143 ~]# 
[root@ceph143 ~]# rbd ls -l -p yinzhengjie-rbd 
NAME      SIZE  PARENT FMT PROT LOCK 
mysqld    5 GiB          2           
wordpress 2 GiB          2           
[root@ceph143 ~]# 


    2 查看镜像的详细信息
[root@ceph142 ~]# rbd info -p yinzhengjie-rbd wordpress | grep '^[[:space:]]features'
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
[root@ceph142 ~]# 
[root@ceph142 ~]# rbd info -p yinzhengjie-rbd mysqld | grep "\sfeatures"
    features: layering, exclusive-lock
[root@ceph142 ~]#

3.修改RBD的镜像特性，由于centos 7 Linux内核版本过低(3.10.0-1127.el7.x86_64)，不支持更多的特性

[root@ceph141 ~]# rbd info -p yinzhengjie-rbd wordpress | grep '^[[:space:]]features'
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd feature disable yinzhengjie-rbd/wordpress object-map fast-diff deep-flatten
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd info -p yinzhengjie-rbd wordpress | grep '\sfeatures'
    features: layering, exclusive-lock
[root@ceph141 ~]#

4.客户端映射镜像(块设备)

    1 拷贝ceph的配置文件，主要用于识别ceph集群地址
[root@harbor250 ~]# scp ceph141:/etc/ceph/ceph.conf /etc/ceph/
ceph.conf                                                                                            100%  264   113.6KB/s   00:00    
[root@harbor250 ~]# 
[root@harbor250 ~]# grep mon_host /etc/ceph/ceph.conf 
mon_host = 10.0.0.141,10.0.0.142,10.0.0.143
[root@harbor250 ~]# 

    2 拷贝ceph的认证文件，主要用于ceph集群认证
[root@harbor250 ~]# scp ceph141:/etc/ceph/ceph.client.admin.keyring /etc/ceph/
ceph.client.admin.keyring                                                                            100%  151   111.1KB/s   00:00    
[root@harbor250 ~]# 
[root@harbor250 ~]# grep key /etc/ceph/ceph.client.admin.keyring 
    key = AQDjFrplyvFCDhAApJg111YMIGQ6/F/x/Y+qpQ==
[root@harbor250 ~]# 


    3 添加映射
[root@harbor250 ~]# ll /dev/rbd*
ls: cannot access /dev/rbd*: No such file or directory
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd -p yinzhengjie-rbd map mysqld
/dev/rbd0
[root@harbor250 ~]#  
[root@harbor250 ~]# ll /dev/rbd*
brw-rw---- 1 root disk 252, 0 Feb  1 12:08 /dev/rbd0

/dev/rbd:
total 0
drwxr-xr-x 2 root root 60 Feb  1 12:08 yinzhengjie-rbd
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd -p yinzhengjie-rbd map wordpress
/dev/rbd1
[root@harbor250 ~]# 
[root@harbor250 ~]# ll /dev/rbd*
brw-rw---- 1 root disk 252,  0 Feb  1 12:08 /dev/rbd0
brw-rw---- 1 root disk 252, 16 Feb  1 12:14 /dev/rbd1

/dev/rbd:
total 0
drwxr-xr-x 2 root root 80 Feb  1 12:14 yinzhengjie-rbd
[root@harbor250 ~]#

5.对块设备进行格式化

[root@harbor250 ~]# mkfs.xfs /dev/rbd0 
meta-data=/dev/rbd0              isize=512    agcount=8, agsize=163840 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=1310720, imaxpct=25
         =                       sunit=1024   swidth=1024 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@harbor250 ~]# 
[root@harbor250 ~]# mkfs.ext4 /dev/rbd1
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done                            
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1024 blocks, Stripe width=1024 blocks
131072 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

[root@harbor250 ~]#

6.创建挂载点

[root@harbor250 ~]# mkdir -pv /yinzhengjie/data/{wordpress,mysql}
mkdir: created directory ‘/yinzhengjie/data’
mkdir: created directory ‘/yinzhengjie/data/wordpress’
mkdir: created directory ‘/yinzhengjie/data/mysql’
[root@harbor250 ~]#

7.尝试手动挂载

[root@harbor250 ~]# mount /dev/rbd0 /yinzhengjie/data/mysql
[root@harbor250 ~]# 
[root@harbor250 ~]# mount /dev/rbd1 /yinzhengjie/data/wordpress
[root@harbor250 ~]# 
[root@harbor250 ~]# df -h | grep rbd
/dev/rbd0                5.0G   33M  5.0G   1% /yinzhengjie/data/mysql
/dev/rbd1                2.0G  6.0M  1.8G   1% /yinzhengjie/data/wordpress
[root@harbor250 ~]#

8.配置ceph开机自启动，(做完实验后可以注释掉，以免当你删除存储池后，无法挂载块设备的问题。)

[root@harbor250 ~]# vim /etc/rc.d/rc.local 

...
rbd -p yinzhengjie-rbd map mysqld
rbd -p yinzhengjie-rbd map wordpress
mount /dev/rbd0 /yinzhengjie/data/mysql
mount /dev/rbd1 /yinzhengjie/data/wordpress
[root@harbor250 ~]# 
[root@harbor250 ~]# ll  /etc/rc.d/rc.local 
-rw-r--r-- 1 root root 620 Feb  1 12:21 /etc/rc.d/rc.local
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# chmod +x /etc/rc.d/rc.local 
[root@harbor250 ~]# 
[root@harbor250 ~]# ll  /etc/rc.d/rc.local 
-rwxr-xr-x 1 root root 620 Feb  1 12:21 /etc/rc.d/rc.local
[root@harbor250 ~]# 
[root@harbor250 ~]# df -h | grep rbd
/dev/rbd0                5.0G   33M  5.0G   1% /yinzhengjie/data/mysql
/dev/rbd1                2.0G  6.0M  1.8G   1% /yinzhengjie/data/wordpress
[root@harbor250 ~]# 
[root@harbor250 ~]# umount /yinzhengjie/data/mysql 
[root@harbor250 ~]# 
[root@harbor250 ~]# df -h | grep rbd
/dev/rbd1                2.0G  6.0M  1.8G   1% /yinzhengjie/data/wordpress
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# umount /dev/rbd1 
[root@harbor250 ~]# 
[root@harbor250 ~]# df -h | grep rbd
[root@harbor250 ~]# 
[root@harbor250 ~]#  reboot 
[root@harbor250 ~]#
[root@harbor250 ~]# df -h | grep yinzhengjie
/dev/rbd0                5.0G   33M  5.0G   1% /yinzhengjie/data/mysql
/dev/rbd1                2.0G  6.0M  1.8G   1% /yinzhengjie/data/wordpress
[root@harbor250 ~]#

9.测试RBD块设备的可用性

[root@harbor250 ~]# cp /etc/os-release /yinzhengjie/data/mysql/
[root@harbor250 ~]# 
[root@harbor250 ~]# cp /etc/hosts /yinzhengjie/data/mysql/
[root@harbor250 ~]# 
[root@harbor250 ~]# cp /etc/fstab /yinzhengjie/data/wordpress/
[root@harbor250 ~]# 
[root@harbor250 ~]# ll /yinzhengjie/data/*
/yinzhengjie/data/mysql:
total 8
-rw-r--r-- 1 root root 288 Feb  1 12:24 hosts
-rw-r--r-- 1 root root 393 Feb  1 12:24 os-release

/yinzhengjie/data/wordpress:
total 20
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
[root@harbor250 ~]#

10.卸载设备

[root@harbor250 ~]# umount /yinzhengjie/data/mysql 
[root@harbor250 ~]# 
[root@harbor250 ~]# umount /yinzhengjie/data/wordpress 
[root@harbor250 ~]# 
[root@harbor250 ~]# df | grep  yinzhengjie
[root@harbor250 ~]#

11.其他客户端重新挂载设备观察数据是否丢失

[root@k8s-worker05 ~]# cat /etc/yum.repos.d/ceph.repo 
[Ceph]
name=Ceph packages for 
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/$basearch
gpgcheck=0
[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch
gpgcheck=0
[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/SRPMS
gpgcheck=0

[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# yum -y install ceph-common
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# scp 10.0.0.141:/etc/ceph/ceph.client.admin.keyring /etc/ceph/
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# scp 10.0.0.141:/etc/ceph/ceph.conf /etc/ceph/
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# rbd  map yinzhengjie-rbd/mysqld
/dev/rbd0
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# mount /dev/rbd0 /mnt/
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# ll /mnt/
total 8
-rw-r--r-- 1 root root 288 Feb  1 12:24 hosts
-rw-r--r-- 1 root root 393 Feb  1 12:24 os-release
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# mkdir /opt/yinzhengjie
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 0
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# rbd  map yinzhengjie-rbd/wordpress
/dev/rbd1
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# mount /dev/rbd1 /opt/yinzhengjie/
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 20
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
[root@k8s-worker05 ~]#

12.两个节点不得使用同一个块设备，否则可能会出现数据不一致的问题！各节点应该单独使用块设备。

    1.客户端1操作如下:
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 20
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# cp /etc/sysconfig/network-scripts/ifcfg-ens33 /opt/yinzhengjie/
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# cp /etc/resolv.conf /opt/yinzhengjie
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 28
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
-rw-r--r-- 1 root root   163 Feb  1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
-rw-r--r-- 1 root root    72 Feb  1 12:34 resolv.conf
[root@k8s-worker05 ~]# 


    2.客户端2操作如下:
[root@harbor250 ~]# mount /dev/rbd0 /yinzhengjie/data/mysql/
[root@harbor250 ~]# 
[root@harbor250 ~]# ll /yinzhengjie/data/mysql/
total 8
-rw-r--r-- 1 root root 288 Feb  1 12:24 hosts
-rw-r--r-- 1 root root 393 Feb  1 12:24 os-release
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# mount /dev/rbd1 /yinzhengjie/data/wordpress/
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# ll /yinzhengjie/data/wordpress/
total 28
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
-rw-r--r-- 1 root root   163 Feb  1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
-rw-r--r-- 1 root root    72 Feb  1 12:34 resolv.conf
[root@harbor250 ~]# 

    3.客户端1继续操作如下:
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 28
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
-rw-r--r-- 1 root root   163 Feb  1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
-rw-r--r-- 1 root root    72 Feb  1 12:34 resolv.conf
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# cp /etc/ceph/ceph.conf /opt/yinzhengjie
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 32
-rw-r--r-- 1 root root   264 Feb  1 12:36 ceph.conf
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
-rw-r--r-- 1 root root   163 Feb  1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
-rw-r--r-- 1 root root    72 Feb  1 12:34 resolv.conf
[root@k8s-worker05 ~]# 


    4.客户端2再次验证，不难发现，两个节点数据已经开始不一致啦！为了避免这种情况，不建议两个节点使用同一个块设备。
[root@harbor250 ~]# ll /yinzhengjie/data -R
/yinzhengjie/data:
total 4
drwxr-xr-x 2 root root   37 Feb  1 12:24 mysql
drwxr-xr-x 3 root root 4096 Feb  1 12:34 wordpress

/yinzhengjie/data/mysql:
total 8
-rw-r--r-- 1 root root 288 Feb  1 12:24 hosts
-rw-r--r-- 1 root root 393 Feb  1 12:24 os-release

/yinzhengjie/data/wordpress:
total 28
-rw-r--r-- 1 root root   465 Feb  1 12:24 fstab
-rw-r--r-- 1 root root   163 Feb  1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb  1 12:17 lost+found
-rw-r--r-- 1 root root    72 Feb  1 12:34 resolv.conf

/yinzhengjie/data/wordpress/lost+found:
total 0
[root@harbor250 ~]#

13.温馨提示

[root@k8s-worker05 ~]# cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# uname -r
4.19.12-1.el7.elrepo.x86_64
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# 
[root@k8s-worker05 ~]# rbd map yinzhengjie-rbd/k8s  # 尽管将内核升级为4系列的，发现依旧不好使，建议将内核升级为5系列的。
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable yinzhengjie-rbd/k8s object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
[root@k8s-worker05 ~]#

三. Ubuntu环境ceph客户端使用rbd实战

1.Ubuntu安装ceph环境

[root@prometheus-server32 ~]# wget -q -O- 'https://download.ceph.com/keys/release.asc' |  apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# echo deb https://download.ceph.com/debian-reef/ $(lsb_release -sc) main | tee /etc/apt/sources.list.d/ceph.list
deb https://download.ceph.com/debian-reef/ jammy main
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# apt update
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# apt -y install ceph-common
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# ceph --version
ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
[root@prometheus-server32 ~]# 


温馨提示:
    但是官方对于Ceph的Nautilus(V14.2.22)版本而言，最高支持的Ubuntu系统是: Ubuntu 20.04 LTS Focal ，最高支持的CentOS 7。
    由于我的系统是"Ubuntu 22.04 LTS Fammy"，不支持Nautilus(V14.2.22)版本安装。 
    因此我安装的Ceph reef(V18.2.1)版本，如果想要安装和我们自定义的Nautilus(V14.2.22)版本。需要降低Ubuntu系统的版本号并做如下调整。

[root@prometheus-server32 ~]# wget -q -O- 'https://download.ceph.com/keys/release.asc' |  apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# echo deb https://download.ceph.com/debian-nautilus/ $(lsb_release -sc) main | tee /etc/apt/sources.list.d/ceph.list
deb https://download.ceph.com/debian-nautilus/ jammy main
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# cat /etc/apt/sources.list.d/ceph.list 
deb https://download.ceph.com/debian-nautilus/ jammy main
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# apt update


    如果网速慢的同学，可以将"download.ceph.com"更换为"mirrors.aliyun.com"以提高速度。



参考链接:
    https://docs.ceph.com/en/latest/install/get-packages/#apt
    https://docs.ceph.com/en/latest/install/get-packages/#debian-packages

2.Ubuntu拷贝ceph文件及证书

[root@prometheus-server32 ~]# scp 10.0.0.141:/etc/ceph/ceph.c* /etc/ceph/
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# ceph -s
  cluster:
    id:     5821e29c-326d-434d-a5b6-c492527eeaad
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 21h)
    mgr: ceph141(active, since 21h), standbys: ceph143, ceph142
    osd: 7 osds: 7 up (since 21h), 7 in (since 21h)

  data:
    pools:   2 pools, 32 pgs
    objects: 43 objects, 86 MiB
    usage:   7.7 GiB used, 1.9 TiB / 2.0 TiB avail
    pgs:     32 active+clean

[root@prometheus-server32 ~]#

3.创建镜像设备

[root@prometheus-server32 ~]# rbd create -s 5G yinzhengjie-rbd/k8s 
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# rbd ls -p yinzhengjie-rbd
k8s
mysqld
wordpress
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# rbd info yinzhengjie-rbd/k8s 
rbd image 'k8s':
    size 5 GiB in 1280 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: 13fa4379e5c8
    block_name_prefix: rbd_data.13fa4379e5c8
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    op_features: 
    flags: 
    create_timestamp: Thu Feb  1 07:21:28 2024
    access_timestamp: Thu Feb  1 07:21:28 2024
    modify_timestamp: Thu Feb  1 07:21:28 2024
[root@prometheus-server32 ~]#

4.Ubuntu系统添加映射

[root@prometheus-server32 ~]# uname -r
5.15.0-92-generic
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# rbd info yinzhengjie-rbd/k8s | grep '\sfeatures'
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# rbd showmapped
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# rbd  map yinzhengjie-rbd/k8s
/dev/rbd0
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# rbd showmapped    # 查看本地的挂载信息。
id  pool             namespace  image  snap  device   
0   yinzhengjie-rbd             k8s    -     /dev/rbd0
[root@prometheus-server32 ~]#

5.对设备进行格式化并挂载

[root@prometheus-server32 ~]# mkfs.xfs  /dev/rbd0 
meta-data=/dev/rbd0              isize=512    agcount=8, agsize=163840 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=1310720, imaxpct=25
         =                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...Done.
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# mount /dev/rbd0 /mnt/
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# df -h | grep mnt
/dev/rbd0                          5.0G   69M  5.0G   2% /mnt
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# cp /etc/os-release /etc/hosts /mnt/
[root@prometheus-server32 ~]# 
[root@prometheus-server32 ~]# ll /mnt/
total 12
drwxr-xr-x  2 root root   37 Feb  1 07:25 ./
drwxr-xr-x 20 root root 4096 Jan 25 02:55 ../
-rw-r--r--  1 root root  226 Feb  1 07:25 hosts
-rw-r--r--  1 root root  386 Feb  1 07:25 os-release
[root@prometheus-server32 ~]#

6.卸载设备

[root@prometheus-server32 ~]# umount /mnt

7.centos系统尝试挂载设备并验证数据

[root@harbor250 ~]# uname -r
3.10.0-1127.el7.x86_64
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd  map yinzhengjie-rbd/k8s  # 发现挂载会失败，报错所不支持特性
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable yinzhengjie-rbd/k8s object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd info yinzhengjie-rbd/k8s | grep '\sfeatures'
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd feature disable yinzhengjie-rbd/k8s object-map fast-diff deep-flatten  # 根据上面的报错禁用特性
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd info yinzhengjie-rbd/k8s | grep '\sfeatures'
    features: layering, exclusive-lock
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd  map yinzhengjie-rbd/k8s  # 禁用特性后，发现可以映射了，映射的设备类型为"/dev/rbd2"
/dev/rbd2
[root@harbor250 ~]# 
[root@harbor250 ~]# mount /dev/rbd2 /mnt/  # 很明显，挂载失败了，说是不支持块设备。
mount: wrong fs type, bad option, bad superblock on /dev/rbd2,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
[root@harbor250 ~]# 
[root@harbor250 ~]# mkfs.xfs /dev/rbd2   # 此处我重新格式化，发现说已经存在了xfs系统，因此需要强行格式化
mkfs.xfs: /dev/rbd2 appears to contain an existing filesystem (xfs).
mkfs.xfs: Use the -f option to force overwrite.
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# mkfs.xfs /dev/rbd2  -f  # 强行格式化后，数据会丢失
meta-data=/dev/rbd2              isize=512    agcount=8, agsize=163840 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=1310720, imaxpct=25
         =                       sunit=1024   swidth=1024 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@harbor250 ~]# 
[root@harbor250 ~]# mount /dev/rbd2 /mnt/  # 格式化后再次挂载，发现数据丢失了。
[root@harbor250 ~]# 
[root@harbor250 ~]# ll /mnt/
total 0
[root@harbor250 ~]# 
[root@harbor250 ~]# cp /etc/sysconfig/network-scripts/ifcfg-ens33 /mnt/   # 添加数据
[root@harbor250 ~]# 
[root@harbor250 ~]# ll /mnt/
total 4
-rw-r--r-- 1 root root 147 Feb  1 15:32 ifcfg-ens33
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# cat /mnt/ifcfg-ens33 
TYPE="Ethernet"
BOOTPROTO="static"
NAME="ens33"
DEVICE="ens33"
ONBOOT="yes"
IPADDR="10.0.0.250"
PREFIX="24"
GATEWAY="10.0.0.254"
DNS1="10.0.0.254"
[root@harbor250 ~]# 
[root@harbor250 ~]# 
[root@harbor250 ~]# umount /mnt 
[root@harbor250 ~]# 
[root@harbor250 ~]# rbd  unmap yinzhengjie-rbd/k8s  # 取消映射
[root@harbor250 ~]#
文章标签：
容器服务Kubernetes版
云数据库 Tair（兼容 Redis）
云数据库 RDS MySQL 版
可观测监控 Prometheus 版
Linux
存储
Ubuntu
相关实践学习
通过Ingress进行灰度发布
本场景您将运行一个简单的应用，部署一个新的应用用于新的发布，并通过Ingress能力实现灰度发布。
容器应用与集群管理
欢迎来到《容器应用与集群管理》课程，本课程是“云原生容器Clouder认证“系列中的第二阶段。课程将向您介绍与容器集群相关的概念和技术，这些概念和技术可以帮助您了解阿里云容器服务ACK/ACK Serverless的使用。同时，本课程也会向您介绍可以采取的工具、方法和可操作步骤，以帮助您了解如何基于容器服务ACK Serverless构建和管理企业级应用。 学习完本课程后，您将能够： 掌握容器集群、容器编排的基本概念 掌握Kubernetes的基础概念及核心思想 掌握阿里云容器服务ACK/ACK Serverless概念及使用方法 基于容器服务ACK Serverless搭建和管理企业级网站应用
尹正杰
目录
相关文章
尹正杰
|
2月前
RBD的块设备的镜像快照实战指南
这篇文章提供了一个关于如何在Ceph RBD块设备上创建和管理快照的详细指南，包括快照的创建、回滚、分层克隆以及快照数量限制的操作步骤。
尹正杰
71 1
shudiFu
|
4月前
|
存储 缓存 关系型数据库
RBD与Cephfs
RBD与Cephfs
shudiFu
85 6
shudiFu
|
4月前
|
存储 Kubernetes 应用服务中间件
k8s使用rbd作为存储
k8s使用rbd作为存储
shudiFu
61 6
qq_734449600
|
11月前
|
存储
qemu 使用 ceph rbd 存储块
qemu 使用 ceph rbd 存储块
qq_734449600
191 0
DevOps和k8s全栈技术
|
Kubernetes 容器
kubernetes挂载ceph rbd和cephfs
kubernetes挂载ceph rbd和cephfs
DevOps和k8s全栈技术
201 0
blackpiglet
|
存储 Linux 块存储
Linux 远程挂载 Ceph RBD 磁盘
block storage RBD 是 Ceph 的块存储方案，最近需要在一台 Linux 服务器上挂载 Ceph 镜像，做法和 Kubernetes 挂 RBD 很像，但部分操作由于 Kubernetes 在镜像中已经固化了，所以将这次完全自己控制的步骤记录下来，加深对 Ceph 挂载的理解。
blackpiglet
2778 0
blackpiglet
|
存储 Kubernetes 文件存储
CephFS 使用
Ceph 之前介绍了 RBD 的使用方法，有了 RBD，远程磁盘挂载的问题就解决了，但 RBD 的问题是不能多个主机共享一个磁盘，如果有一份数据很多客户端都要读写该怎么办呢？这时 CephFS 作为文件系统存储解决方案就派上用场了。
blackpiglet
2364 0
技术小大人
|
5G 块存储
Ceph集群块设备使用-创建和使用OSD
技术小大人
943 0
技术小大人
|
存储 Linux 块存储
CentOS7 Ceph RBD CephFS 存储
技术小大人
1669 0
余二五
|
存储 网络协议 数据安全/隐私保护
glusterfs——volume管理
余二五
1405 0