存储系统是个专业性的整个体系，我们只用常见方式搭建，具体优化需要参照公司自己的产品等各种进行调整

一、Ceph

https://ceph.io/

1、基本概念

Ceph可以有

Ceph对象存储：键值存储，其接口就是简单的GET,PUT,DEL等。如七牛，阿里云oss等
Ceph块设备：AWS的EBS，青云的云硬盘和阿里云的盘古系统，还有Ceph的RBD(RBD是Ceph面向块存储的接口)
Ceph文件系统：它比块存储具有更丰富的接口，需要考虑目录、文件属性等支持，实现一个支持并行化的文件存储应该是最困难的。

一个Ceph存储集群需要

至少一个Ceph监视器、Ceph管理器、Ceph OSD（对象存储守护程序）
需要运行Ceph文件系统客户端，则需要部署 Ceph Metadata Server。

Monitors: Ceph Monitor (ceph-mon) 监视器：维护集群状态信息

维护集群状态的映射，包括监视器映射，管理器映射，OSD映射，MDS映射和CRUSH映射。
这些映射是Ceph守护程序相互协调所必需的关键群集状态。
监视器还负责管理守护程序和客户端之间的身份验证。
通常至少需要三个监视器才能实现冗余和高可用性。

Managers: Ceph Manager 守护进程(ceph-mgr) : 负责跟踪运行时指标和Ceph集群的当前状态
Ceph Manager守护进程（ceph-mgr）负责跟踪运行时指标和Ceph集群的当前状态
包括存储利用率，当前性能指标和系统负载。
Ceph Manager守护程序还托管基于python的模块，以管理和公开Ceph集群信息，包括基于Web的Ceph Dashboard和REST API。
通常，至少需要两个管理器才能实现高可用性。
Ceph OSDs: Ceph OSD (对象存储守护进程, ceph-osd) 【存储数据】
通过检查其他Ceph OSD守护程序的心跳来存储数据，处理数据复制，恢复，重新平衡，并向Ceph监视器和管理器提供一些监视信息。
通常至少需要3个Ceph OSD才能实现冗余和高可用性。
MDSs: Ceph Metadata Server (MDS, ceph-mdsceph元数据服务器)
存储能代表 Ceph File System 的元数据(如：Ceph块设备和Ceph对象存储不使用MDS).
Ceph元数据服务器允许POSIX文件系统用户执行基本命令（如ls，find等），而不会给Ceph存储集群带来巨大负担

二、Rook

1、基本概念

Rook是云原生平台的存储编排工具

Rook工作原理如下：

Rook架构如下

RGW：为Restapi Gateway

2、operator是什么

k8s中operator+CRD（CustomResourceDefinitions【k8s自定义资源类型】），可以快速帮我们部署一些有状态应用集群，如redis，mysql，Zookeeper等。

Rook的operator是我们k8s集群和存储集群之间进行交互的解析器

CRD：CustomResourceDefinitions (自定义资源)；如：Itdachang

operator：这个能处理自定义资源类型

三、部署

https://rook.io/docs/rook/v1.6/ceph-quickstart.html

1、查看前提条件

Raw devices (no partitions or formatted filesystems)；原始磁盘，无分区或者格式化
Raw partitions (no formatted filesystem)；原始分区，无格式化文件系统

fdisk -l

找到自己挂载的磁盘

如： /dev/sda

# VBoxManage list vms

"k8s-master1_node-4_1653898385983_84970" {89e9bcaf-9a87-4a5a-9560-4ca6984de5a6}

"k8s-master1_master_1653898563856_45832" {2405b61f-0ca1-48fd-8c0f-c578ca040377}

"k8s-master2_master_1653898922131_7237" {27584449-5ac1-4244-9730-770d2af03b41}

"k8s-master3_master_1653957843986_74361" {f4ab072b-a8ab-4e80-8907-3d84b67d7f12}

"k8s-node1_node_1653959160909_59105" {6aab8e99-73ac-48bd-943e-bd0ca399ada0}

"k8s-node2_node_1653959211808_83412" {c3cc440c-68c0-466f-a842-80907cce941a}

"k8s-node3_node_1653959261692_11714" {16909485-56f3-434d-8297-199e978aeb0c}

VirtualBox添加硬盘

--2022年6月8日14:20:47 wx

vagrant 虚拟机上安装ceph

# 创建虚拟卷，并指定大小和位置

# vdi 代表virtual box原生的

# vmdk 代表VMware开发的但支持virtualbox....

VBoxManage createvdi --filename "/gpmaster/gpseg-1_down/VirtualBox VMs/k8s-node1_node_1653959160909_59105/k8s-node1-50g.vdi" --size 51200

VBoxManage storagectl k8s-node1_node_1653959160909_59105 --name SATA --add sata --controller IntelAhci --bootable on

# 将第一步创建的虚拟存储卷绑定到指定虚拟机上

VBoxManage storageattach k8s-node1_node_1653959160909_59105 --storagectl SATA --port 0 --device 0 --type hdd --medium "/gpmaster/gpseg-1_down/VirtualBox VMs/k8s-node1_node_1653959160909_59105/k8s-node1-50g.vdi"

VBoxManage createvdi --filename "/gpmaster/gpseg-1_down/VirtualBox VMs/k8s-node3_node_1653959261692_11714/k8s-node3-50g.vdi" --size 51200

VBoxManage storagectl k8s-node3_node_1653959261692_11714 --name SATA --add sata --controller IntelAhci --bootable on

VBoxManage storageattach k8s-node3_node_1653959261692_11714 --storagectl SATA --port 0 --device 0 --type hdd --medium "/gpmaster/gpseg-1_down/VirtualBox VMs/k8s-node3_node_1653959261692_11714/k8s-node3-50g.vdi"

报错处理

VBoxManage: error: Could not find a controller named 'SATA'

创建存储控制器IDE、SATA

VBoxManage storagectl k8s-node1_node_1653959160909_59105 --name SATA --add sata --controller IntelAhci --bootable on

解除关联：

VBoxManage storageattach UbuntuRDHome --storagectl SATA --port 0 --device 0 --type hdd --medium none

VBoxManage storageattach UbuntuRDHome --storagectl SATA --port 1 --device 0 --type hdd --medium none

移除控制器：

VBoxManage storagectl UbuntuRDHome --name IDE --remove

VBoxManage storagectl UbuntuRDHome --name SATA --remove

参考: https://blog.csdn.net/vevenlcf/article/details/82114788

[root@node3 ~]# lsblk -f

NAME FSTYPE LABEL UUID MOUNTPOINT

sda

└─sda1 xfs 1c419d6c-5064-4a2b-953c-05b2c67edb15 /

sdb

2、部署&修改operator

cd cluster/examples/kubernetes/ceph

kubectl create -f crds.yaml -f common.yaml -f operator.yaml #注意修改operator镜像

# verify the rook-ceph-operator is in the `Running` state before proceeding

kubectl -n rook-ceph get pod

修改operator.yaml

把以前的默认镜像换成能用的，如下

# ROOK_CSI_CEPH_IMAGE: "registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/cephcsi:v3.3.1"

# ROOK_CSI_REGISTRAR_IMAGE: "registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/csi-node-driver-registrar:v2.0.1"

# ROOK_CSI_RESIZER_IMAGE: "registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/csi-resizer:v1.0.1"

# ROOK_CSI_PROVISIONER_IMAGE: "registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/csi-provisioner:v2.0.4"

# ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/csi-snapshotter:v4.0.0"

# ROOK_CSI_ATTACHER_IMAGE: "registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/csi-attacher:v3.0.2"

镜像： rook/ceph:v1.6.3 换成 registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/rook-ceph:v1.6.3

## 建议修改以下的东西。在operator.yaml里面

# ROOK_CSI_CEPH_IMAGE: "10.50.10.185/ceph/quay.io/cephcsi/cephcsi:v3.3.1"

# ROOK_CSI_REGISTRAR_IMAGE: "10.50.10.185/ceph/k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0"

# ROOK_CSI_RESIZER_IMAGE: "10.50.10.185/ceph/k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"

# ROOK_CSI_PROVISIONER_IMAGE: "10.50.10.185/ceph/k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2"

# ROOK_CSI_SNAPSHOTTER_IMAGE: "10.50.10.185/ceph/k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1"

# ROOK_CSI_ATTACHER_IMAGE: "10.50.10.185/ceph/k8s.gcr.io/sig-storage/csi-attacher:v3.2.1"

更换镜像脚本

-- cluster.yaml

"ceph/ceph:v15.2.13",

-- operator.yaml

"quay.io/cephcsi/cephcsi:v3.3.1",

"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0",

"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0",

"k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2",

"k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1",

"k8s.gcr.io/sig-storage/csi-attacher:v3.2.1"

3、部署集群

修改cluster.yaml使用我们指定的磁盘当做存储节点即可

storage: # cluster level storage configuration and selection

useAllNodes: false

useAllDevices: false

config:

osdsPerDevice: "3" #每个设备osd数量

nodes:

- name: "node3"

devices:

- name: "sdb"

- name: "node1"

devices:

- name: "sdb"

- name: "node2"

devices:

- name: "sdb"

镜像： ceph/ceph:v15.2.11 换成 10.50.10.185/ceph/ceph/ceph:v15.2.13

4、部署dashboard

https://www.rook.io/docs/rook/v1.6/ceph-dashboard.html

前面的步骤，已经自动部署了。

kubectl -n rook-ceph get service

#查看service

#为了方便访问我们改为nodePort。应用nodePort文件

#获取访问密码

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

#默认账号 admin

4/qt]e5wad_HY:0&V.ba

MGR: 2；只有一个能用

Mon: 3; 全部可用

Osd: 全部可用

部署了高可用的mgr：提供页面。

先部署nodePort，验证到底哪个mgr不能访问

他是主备模式的高可用。kubectl get svc -n rook-ceph|grep dashboard

curl 访问dashboard确定哪个mgr不能访问

自己做一个可访问到的service。如果哪天服务器故障mgr有问题。修改svc的selector即可

apiVersion: v1

kind: Service

metadata:

labels:

app: rook-ceph-mgr

ceph_daemon_id: a

rook_cluster: rook-ceph

namespace: rook-ceph

spec:

ports:

- name: dashboard

port: 8443

protocol: TCP

targetPort: 8443

selector: #service选择哪些Pod

app: rook-ceph-mgr

ceph_daemon_id: a

rook_cluster: rook-ceph

sessionAffinity: None

type: ClusterIP

再部署如下的ingress

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

namespace: rook-ceph

annotations:

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

nginx.ingress.kubernetes.io/server-snippet: |

proxy_ssl_verify off;

spec:

# tls: 不用每个名称空间都配置证书信息

# - hosts:

# - itdachang.com

# - 未来的

# secretName: testsecret-tls

rules:

- host: rook.itdachang.com

http:

paths:

- path: /

pathType: Prefix

backend:

service:

port:

number: 8443

5、神坑

#部署完成的最终结果一定要有这些组件

NAME READY STATUS RESTARTS AGE

csi-cephfsplugin-provisioner-d77bb49c6-n5tgs 5/5 Running 0 140s

csi-cephfsplugin-provisioner-d77bb49c6-v9rvn 5/5 Running 0 140s

csi-cephfsplugin-rthrp 3/3 Running 0 140s

csi-rbdplugin-hbsm7 3/3 Running 0 140s

csi-rbdplugin-provisioner-5b5cd64fd-nvk6c 6/6 Running 0 140s

csi-rbdplugin-provisioner-5b5cd64fd-q7bxl 6/6 Running 0 140s

rook-ceph-crashcollector-minikube-5b57b7c5d4-hfldl 1/1 Running 0 105s

rook-ceph-mgr-a-64cd7cdf54-j8b5p 1/1 Running 0 77s

rook-ceph-mon-a-694bb7987d-fp9w7 1/1 Running 0 105s

rook-ceph-mon-b-856fdd5cb9-5h2qk 1/1 Running 0 94s

rook-ceph-mon-c-57545897fc-j576h 1/1 Running 0 85s

rook-ceph-operator-85f5b946bd-s8grz 1/1 Running 0 92m

rook-ceph-osd-0-6bb747b6c5-lnvb6 1/1 Running 0 23s

rook-ceph-osd-1-7f67f9646d-44p7v 1/1 Running 0 24s

rook-ceph-osd-2-6cd4b776ff-v4d68 1/1 Running 0 25s

rook-ceph-osd-prepare-node1-vx2rz 0/2 Completed 0 60s

rook-ceph-osd-prepare-node2-ab3fd 0/2 Completed 0 60s

rook-ceph-osd-prepare-node3-w4xyz 0/2 Completed 0 60s

#解决方案。

#找到自己的operator，删除这个pod，让重新执行

kubectl delete pod rook-ceph-operator-65965c66b5-cxrl8 -n rook-ceph

#rbd：是ceph用来做块存储的

#cephfs：是用来做共享文件系统的

6、卸载

# rook集群的清除，

##1、 delete -f 之前的yaml

##2、再执行如下命令

kubectl -n rook-ceph get cephcluster

kubectl -n rook-ceph patch cephclusters.ceph.rook.io rook-ceph -p '{"metadata":{"finalizers": []}}' --type=merge

##3、清除每个节点的 /var/lib/rook 目录

## 顽固的自定义资源删除；

kubectl -n rook-ceph patch cephblockpool.ceph.rook.io replicapool -p '{"metadata":{"finalizers": []}}' --type=merge

Rook+Ceph；

Rook帮我们创建好 StorageClass

pvc只需要指定存储类，Rook自动调用 StorageClass 里面的 Provisioner供应商，接下来对ceph集群操作

Ceph

Block：块存储。RWO（ReadWriteOnce）单节点读写【一个Pod操作一个自己专属的读写区】，适用于（有状态副本集）

Share FS：共享存储。RWX（ReadWriteMany）多节点读写【多个Pod操作同一个存储区，可读可写】，适用于无状态应用。（文件系统）

......

总结：无状态应用随意复制多少份，一定用到RWX能力。有状态应用复制任意份，每份都只是读写自己的存储，用到RWO（优先）或者RWX。

直接通过Rook可以使用到任意能力的存储。

CRI、CNI、CSI

CRI：Container Runtime Interface：容器运行时接口（k8s集群整合容器运行时环境）

CNI：Container Network Interface：容器网络接口（k8s集群整合网络组件的接口）

CSI：Container Storage Interface：容器存储接口（k8s集群整合存储组件的接口）

kubelet启动一个Pod。CRI、CNI、CSI 起作用的顺序

启动Pod流程

每个Pod，都伴随一个Pause容器（沙箱容器）。真正的容器（nginx-pvc）和沙箱容器是公用一个网络、存储、名称空间。。。。

启动沙箱容器。给沙箱容器设置好网络，存储

CRI。创建沙箱容器的运行时环境

CNI。挂载沙箱容器网络等

CSI。调用存储系统进行数据挂载。（提前把应用容器需要挂载的挂进来）

启动应用容器。（kubectl get pod 1/1【不算沙箱容器】）

应用容器直接创建运行时CRI，用以上的 CNI、CSI

从应用容器角度出发：CSI先与CRI启动。

从Pod出发。CRI。CNI。CSI

沙箱容器代码 https://github.com/kubernetes/kubernetes/blob/d541872f9a036ed4f792232e43fde6dacf0e1084/pkg/kubelet/dockershim/docker_sandbox.go#L89

应用容器 https://github.com/kubernetes/kubernetes/blob/d541872f9a036ed4f792232e43fde6dacf0e1084/pkg/kubelet/kubelet.go#L1469

判断网络（前面准备好的）

挂载

容器启动

报错: Rook Ceph OSD异常，格式化osd硬盘重新挂载

https://www.cnblogs.com/deny/p/14214963.html

四、实战

1、块存储(RDB)

RDB： RADOS Block Devices

RADOS： Reliable, Autonomic Distributed Object Store

不能是RWX模式。

1、配置

RWO:（ReadWriteOnce）

https://www.rook.io/docs/rook/v1.6/ceph-block.html

常用块存储。RWO模式；STS删除，pvc不会删除，需要自己手动维护

apiVersion: ceph.rook.io/v1

kind: CephBlockPool

metadata:

namespace: rook-ceph

spec:

failureDomain: host #容灾模式，host或者osd

replicated:

size: 2 #数据副本数量

---

apiVersion: storage.k8s.io/v1

kind: StorageClass #存储驱动

metadata:

# Change "rook-ceph" provisioner prefix to match the operator namespace if needed

provisioner: rook-ceph.rbd.csi.ceph.com

parameters:

# clusterID is the namespace where the rook cluster is running

clusterID: rook-ceph

# Ceph pool into which the RBD image shall be created

pool: replicapool

# (optional) mapOptions is a comma-separated list of map options.

# For krbd options refer

# https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options

# For nbd options refer

# https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options

# mapOptions: lock_on_read,queue_depth=1024

# (optional) unmapOptions is a comma-separated list of unmap options.

# For krbd options refer

# https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options

# For nbd options refer

# https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options

# unmapOptions: force

# RBD image format. Defaults to "2".

imageFormat: "2"

# RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature.

imageFeatures: layering

# The secrets contain Ceph admin credentials.

csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner

csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph

csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner

csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph

csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node

csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph

# Specify the filesystem type of the volume. If not specified, csi-provisioner

# will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock

# in hyperconverged settings where the volume is mounted on the same node as the osds.

csi.storage.k8s.io/fstype: ext4

# Delete the rbd volume when a PVC is deleted

reclaimPolicy: Delete

allowVolumeExpansion: true

2、STS案例实战

apiVersion: apps/v1

kind: StatefulSet

metadata:

namespace: default

spec:

selector:

matchLabels:

app: sts-nginx # has to match .spec.template.metadata.labels

serviceName: "sts-nginx"

replicas: 3 # by default is 1

template:

metadata:

labels:

app: sts-nginx # has to match .spec.selector.matchLabels

spec:

terminationGracePeriodSeconds: 10

containers:

- name: sts-nginx

image: nginx

ports:

- containerPort: 80

volumeMounts:

- name: www

mountPath: /usr/share/nginx/html

volumeClaimTemplates:

- metadata:

spec:

accessModes: [ "ReadWriteOnce" ]

storageClassName: "rook-ceph-block"

resources:

requests:

storage: 20Mi

---

apiVersion: v1

kind: Service

metadata:

namespace: default

spec:

selector:

app: sts-nginx

type: ClusterIP

ports:

- name: sts-nginx

port: 80

targetPort: 80

protocol: TCP

测试：创建sts、修改nginx数据、删除sts、重新创建sts。他们的数据丢不丢，共享不共享

2、文件存储(CephFS)

1、配置

常用文件存储。 RWX模式；如：10个Pod共同操作一个地方

https://rook.io/docs/rook/v1.6/ceph-filesystem.html

apiVersion: ceph.rook.io/v1

kind: CephFilesystem

metadata:

namespace: rook-ceph # namespace:cluster

spec:

# The metadata pool spec. Must use replication.

metadataPool:

replicated:

size: 3

requireSafeReplicaSize: true

parameters:

# Inline compression mode for the data pool

# Further reference: https://docs.ceph.com/docs/nautilus/rados/configuration/bluestore-config-ref/#inline-compression

compression_mode:

none

# gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool

# for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size

#target_size_ratio: ".5"

# The list of data pool specs. Can use replication or erasure coding.

dataPools:

- failureDomain: host

replicated:

size: 3

# Disallow setting pool with replica 1, this could lead to data loss without recovery.

# Make sure you're *ABSOLUTELY CERTAIN* that is what you want

requireSafeReplicaSize: true

parameters:

# Inline compression mode for the data pool

# Further reference: https://docs.ceph.com/docs/nautilus/rados/configuration/bluestore-config-ref/#inline-compression

compression_mode:

none

# gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool

# for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size

#target_size_ratio: ".5"

# Whether to preserve filesystem after CephFilesystem CRD deletion

preserveFilesystemOnDelete: true

# The metadata service (mds) configuration

metadataServer:

# The number of active MDS instances

activeCount: 1

# Whether each active MDS instance will have an active standby with a warm metadata cache for faster failover.

# If false, standbys will be available, but will not have a warm cache.

activeStandby: true

# The affinity rules to apply to the mds deployment

placement:

# nodeAffinity:

# requiredDuringSchedulingIgnoredDuringExecution:

# nodeSelectorTerms:

# - matchExpressions:

# - key: role

# operator: In

# values:

# - mds-node

# topologySpreadConstraints:

# tolerations:

# - key: mds-node

# operator: Exists

# podAffinity:

podAntiAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

- labelSelector:

matchExpressions:

- key: app

operator: In

values:

- rook-ceph-mds

# topologyKey: kubernetes.io/hostname will place MDS across different hosts

topologyKey: kubernetes.io/hostname

preferredDuringSchedulingIgnoredDuringExecution:

- weight: 100

podAffinityTerm:

labelSelector:

matchExpressions:

- key: app

operator: In

values:

- rook-ceph-mds

# topologyKey: */zone can be used to spread MDS across different AZ

# Use <topologyKey: failure-domain.beta.kubernetes.io/zone> in k8s cluster if your cluster is v1.16 or lower

# Use <topologyKey: topology.kubernetes.io/zone> in k8s cluster is v1.17 or upper

topologyKey: topology.kubernetes.io/zone

# A key/value list of annotations

annotations:

# key: value

# A key/value list of labels

labels:

# key: value

resources:

# The requests and limits set here, allow the filesystem MDS Pod(s) to use half of one CPU core and 1 gigabyte of memory

# limits:

# cpu: "500m"

# memory: "1024Mi"

# requests:

# cpu: "500m"

# memory: "1024Mi"

# priorityClassName: my-priority-class

mirroring:

enabled: false

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

# Change "rook-ceph" provisioner prefix to match the operator namespace if needed

provisioner: rook-ceph.cephfs.csi.ceph.com

parameters:

# clusterID is the namespace where operator is deployed.

clusterID: rook-ceph

# CephFS filesystem name into which the volume shall be created

fsName: myfs

# Ceph pool into which the volume shall be created

# Required for provisionVolume: "true"

pool: myfs-data0

# The secrets contain Ceph admin credentials. These are generated automatically by the operator

# in the same namespace as the cluster.

csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner

csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph

csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner

csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph

csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node

csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph

reclaimPolicy: Delete

allowVolumeExpansion: true

2、测试

apiVersion: apps/v1

kind: Deployment

metadata:

namespace: default

labels:

app: nginx-deploy

spec:

selector:

matchLabels:

app: nginx-deploy

replicas: 3

strategy:

rollingUpdate:

maxSurge: 25%

maxUnavailable: 25%

type: RollingUpdate

template:

metadata:

labels:

app: nginx-deploy

spec:

containers:

- name: nginx-deploy

image: nginx

volumeMounts:

- name: localtime

mountPath: /etc/localtime

- name: nginx-html-storage

mountPath: /usr/share/nginx/html

volumes:

- name: localtime

hostPath:

path: /usr/share/zoneinfo/Asia/Shanghai

- name: nginx-html-storage

persistentVolumeClaim:

claimName: nginx-pv-claim

---

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

labels:

app: nginx-deploy

spec:

storageClassName: rook-cephfs

accessModes:

- ReadWriteMany ##如果是ReadWriteOnce将会是什么效果

resources:

requests:

storage: 10Mi

测试，创建deploy、修改页面、删除deploy，新建deploy是否绑定成功，数据是否在。

3、pvc扩容

参照CSI（容器存储接口）文档：

卷扩容：https://www.rook.io/docs/rook/v1.6/ceph-csi-drivers.html#dynamically-expand-volume

1、动态卷扩容

# 之前创建storageclass的时候已经配置好了

# 测试：去容器挂载目录 curl -O 某个大文件默认不能下载

#修改原来的PVC，可以扩充容器。

#注意，只能扩容，不能缩容

有状态应用（3个副本）使用块存储。自己操作自己的pvc挂载的pv；也不丢失

无状态应用（3个副本）使用共享存储。很多人操作一个pvc挂载的一个pv；也不丢失

其他Pod可以对数据进行修改

MySQL 有状态做成主节点。。。MySQL - Master ---- pv

MySQL 无状态只读挂载master的 pvc。

K8s部署分布式存储Ceph系统搭建与实战

一、Ceph

1、基本概念

二、Rook

1、基本概念

2、operator是什么

三、部署

1、查看前提条件

2、部署&修改operator

3、部署集群

4、部署dashboard

5、神坑

6、卸载

四、实战

1、块存储(RDB)

1、配置

2、STS案例实战

2、文件存储(CephFS)

1、配置

2、测试

3、pvc扩容

1、动态卷扩容

4、更多参照官方文档

热门文章

最新文章

相关课程

相关电子书

相关实验场景

推荐镜像