云原生|kubernetes|持久化存储pv，pvc和StorageClass的学习

[root@master coredns]# k explain pv.spec
KIND:     PersistentVolume
VERSION:  v1
RESOURCE: spec <Object>
DESCRIPTION:
     Spec defines a specification of a persistent volume owned by the cluster.
     Provisioned by an administrator. More info:
     https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes
     PersistentVolumeSpec is the specification of a persistent volume.
FIELDS:
   accessModes  <[]string>
     AccessModes contains all ways the volume can be mounted. More info:
     https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes
   awsElasticBlockStore <Object>
     AWSElasticBlockStore represents an AWS Disk resource that is attached to a
     kubelet's host machine and then exposed to the pod. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
   azureDisk  <Object>
     AzureDisk represents an Azure Data Disk mount on the host and bind mount to
     the pod.
   azureFile  <Object>
     AzureFile represents an Azure File Service mount on the host and bind mount
     to the pod.
   capacity <map[string]string>
     A description of the persistent volume's resources and capacity. More info:
     https://kubernetes.io/docs/concepts/storage/persistent-volumes#capacity
   cephfs <Object>
     CephFS represents a Ceph FS mount on the host that shares a pod's lifetime
   cinder <Object>
     Cinder represents a cinder volume attached and mounted on kubelets host
     machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
   claimRef <Object>
     ClaimRef is part of a bi-directional binding between PersistentVolume and
     PersistentVolumeClaim. Expected to be non-nil when bound. claim.VolumeName
     is the authoritative bind between PV and PVC. More info:
     https://kubernetes.io/docs/concepts/storage/persistent-volumes#binding
   csi  <Object>
     CSI represents storage that is handled by an external CSI driver (Beta
     feature).
   fc <Object>
     FC represents a Fibre Channel resource that is attached to a kubelet's host
     machine and then exposed to the pod.
   flexVolume <Object>
     FlexVolume represents a generic volume resource that is
     provisioned/attached using an exec based plugin.
   flocker  <Object>
     Flocker represents a Flocker volume attached to a kubelet's host machine
     and exposed to the pod for its usage. This depends on the Flocker control
     service being running
   gcePersistentDisk  <Object>
     GCEPersistentDisk represents a GCE Disk resource that is attached to a
     kubelet's host machine and then exposed to the pod. Provisioned by an
     admin. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
   glusterfs  <Object>
     Glusterfs represents a Glusterfs volume that is attached to a host and
     exposed to the pod. Provisioned by an admin. More info:
     https://examples.k8s.io/volumes/glusterfs/README.md
   hostPath <Object>
     HostPath represents a directory on the host. Provisioned by a developer or
     tester. This is useful for single-node development and testing only!
     On-host storage is not supported in any way and WILL NOT WORK in a
     multi-node cluster. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#hostpath
   iscsi  <Object>
     ISCSI represents an ISCSI Disk resource that is attached to a kubelet's
     host machine and then exposed to the pod. Provisioned by an admin.
   local  <Object>
     Local represents directly-attached storage with node affinity
   mountOptions <[]string>
     A list of mount options, e.g. ["ro", "soft"]. Not validated - mount will
     simply fail if one is invalid. More info:
     https://kubernetes.io/docs/concepts/storage/persistent-volumes/#mount-options
   nfs  <Object>
     NFS represents an NFS mount on the host. Provisioned by an admin. More
     info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
   nodeAffinity <Object>
     NodeAffinity defines constraints that limit what nodes this volume can be
     accessed from. This field influences the scheduling of pods that use this
     volume.
   persistentVolumeReclaimPolicy  <string>
     What happens to a persistent volume when released from its claim. Valid
     options are Retain (default for manually created PersistentVolumes), Delete
     (default for dynamically provisioned PersistentVolumes), and Recycle
     (deprecated). Recycle must be supported by the volume plugin underlying
     this PersistentVolume. More info:
     https://kubernetes.io/docs/concepts/storage/persistent-volumes#reclaiming
   photonPersistentDisk <Object>
     PhotonPersistentDisk represents a PhotonController persistent disk attached
     and mounted on kubelets host machine
   portworxVolume <Object>
     PortworxVolume represents a portworx volume attached and mounted on
     kubelets host machine
   quobyte  <Object>
     Quobyte represents a Quobyte mount on the host that shares a pod's lifetime
   rbd  <Object>
     RBD represents a Rados Block Device mount on the host that shares a pod's
     lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md
   scaleIO  <Object>
     ScaleIO represents a ScaleIO persistent volume attached and mounted on
     Kubernetes nodes.
   storageClassName <string>
     Name of StorageClass to which this persistent volume belongs. Empty value
     means that this volume does not belong to any StorageClass.
   storageos  <Object>
     StorageOS represents a StorageOS volume that is attached to the kubelet's
     host machine and mounted into the pod More info:
     https://examples.k8s.io/volumes/storageos/README.md
   volumeMode <string>
     volumeMode defines if a volume is intended to be used with a formatted
     filesystem or to remain in raw block state. Value of Filesystem is implied
     when not included in spec.
   vsphereVolume  <Object>
     VsphereVolume represents a vSphere volume attached and mounted on kubelets
     host machine

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  strategy: {}
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: {}
        volumeMounts:
        - name: cache-volume
          mountPath: /cache
      volumes:
      - name: cache-volume
        emptyDir:
          medium:
          sizeLimit: 128Mi

[root@master ~]# k exec -it  nginx-f9df8c74f-gf9vz -- /bin/bash
root@nginx-f9df8c74f-gf9vz:/# cd cache/
root@nginx-f9df8c74f-gf9vz:/cache# echo "hello world " >test.fiel
root@nginx-f9df8c74f-gf9vz:/cache# 
exit

[root@slave1 ~]# find / -name test.fiel
/var/lib/kubelet/pods/db7976ed-4c86-4dae-ba2a-92d4dbc7b1e0/volumes/kubernetes.io~empty-dir/cache-volume/test.fiel
[root@slave1 ~]# cat /var/lib/kubelet/pods/db7976ed-4c86-4dae-ba2a-92d4dbc7b1e0/volumes/kubernetes.io~empty-dir/cache-volume/test.fiel
hello world 

[root@slave1 ~]# cat /var/lib/kubelet/pods/db7976ed-4c86-4dae-ba2a-92d4dbc7b1e0/volumes/kubernetes.io~empty-dir/cache-volume/test.fiel
cat: /var/lib/kubelet/pods/db7976ed-4c86-4dae-ba2a-92d4dbc7b1e0/volumes/kubernetes.io~empty-dir/cache-volume/test.fiel: No such file or directory

[root@master ~]# k explain deploy.spec.template.spec.volumes.hostPath
KIND:     Deployment
VERSION:  apps/v1
RESOURCE: hostPath <Object>
DESCRIPTION:
     HostPath represents a pre-existing file or directory on the host machine
     that is directly exposed to the container. This is generally used for
     system agents or other privileged things that are allowed to see the host
     machine. Most containers will NOT need this. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#hostpath
     Represents a host path mapped into a pod. Host path volumes do not support
     ownership management or SELinux relabeling.
FIELDS:
   path <string> -required-
     Path of the directory on the host. If the path is a symlink, it will follow
     the link to the real path. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#hostpath
   type <string>
     Type for HostPath Volume Defaults to "" More info:
     https://kubernetes.io/docs/concepts/storage/volumes#hostpath

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  strategy: {}
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: {}
        volumeMounts:
        - mountPath: /var/local/aaa
          name: mydir
        - mountPath: /var/local/aaa/1.txt
          name: myfile
      volumes:
      - name: mydir
        hostPath:
          path: /var/local/aaa
          type: DirectoryOrCreate
      - name: myfile
        hostPath:
          path: /var/local/aaa/1.txt
          type: FileOrCreate

[root@slave1 aaa]# pwd
/var/local/aaa
[root@slave1 aaa]# ls
1.txt

apiVersion: v1
kind: Namespace
metadata:
  name: web
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nginx-pv
  namespace: web
  labels:
    type: local
spec:
  capacity:
    storage: 3Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
#  storageClassName: local-storage
  local:
    path: "/opt/nginx/data"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key:  kubernetes.io/hostname
          operator: In
          values:
          - k8s-node2

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-pvc
  namespace: web
  labels:
    app: nginx-pvc
spec:
  accessModes:
  - ReadWriteOnce #此处需要和pv对应才能匹配
  resources:
    requests:
      storage: 2Gi
#  storageClassName: local-storage #此处需要和StorageClass.yaml匹配

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: web
spec:
  containers:
  - name: nginx
    image: nginx:1.18
    volumeMounts:
    - name: nginx-persistent-storage
      mountPath: "/usr/share/nginx/html" #不需要修改，映射到镜像内部目录
  volumes:
    - name: nginx-persistent-storage
      persistentVolumeClaim:
        claimName: nginx-pvc #对应到pvc的名字
  tolerations:
  - operator: Exists
    effect: NoExecute
  nodeName: k8s-node2
  affinity:  #亲和性设置
    nodeAffinity: #申明是node亲和策略
      requiredDuringSchedulingIgnoredDuringExecution: # 硬限制
        nodeSelectorTerms:
        - matchExpressions: # 匹配env的值在["xxx","yyy"]中的标签，实际没有设置此标签，所以会匹配失败
          - key: node
            operator: In
            values: ["web","dev"]

yum install nfs rpcbind nfs-utils -y

[root@slave2 local]# cat /etc/exports
/data/nfs-sc  10.244.0.0/16(rw,no_root_squash,no_subtree_check) 192.168.217.16(rw,no_root_squash,no_subtree_check) 192.168.217.0/24(rw,no_root_squash,no_subtree_check) *(rw,no_root_squash,subtree_check)
systemctl enable nfs rpcbind
systemctl restart nfs rpcbind
验证：
[root@slave2 local]# showmount -e 192.168.217.18
Export list for 192.168.217.18:
/data/nfs-sc (everyone)

nfs存储卷可以将现有的nfs-server上的存储空间挂载到Pod中使用。当删除Pod时，nfs存储卷的内容会被保留，卷仅是被卸载而不是删除。而且NFS是文件系统级共享服务，支持被多个Pod挂载使用。定义NFS存储卷时支持嵌套使用3个字段：
server：NFS Server的地址或域名
path：NFS Server共享的目录
readOnly：是否以只读方式挂载，默认false

apiVersion: apps/v1
kind: Deployment
metadata:
  name: volume-nfs-demo
  namespace: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: volume-nfs-demo
  template:
    metadata:
      labels:
        app: volume-nfs-demo
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - name: nfs-volume
          mountPath: /usr/share/nginx/html/mysite
      volumes:
      - name: nfs-volume
        nfs:
         server: 192.168.217.18
         path: /data/nfs-sc

[root@slave2 nfs-sc]# echo "192.168.217.18" >index.html
[root@slave2 nfs-sc]# pwd
/data/nfs-sc

[root@master nginx]# k exec -it volume-nfs-demo-6dd859c5fc-8ntzs -n web -- /bin/bash
root@volume-nfs-demo-6dd859c5fc-8ntzs:/# curl localhost/mysite/
192.168.217.18

[root@slave2 nfs-sc]# mkdir -p /data/nfs-sc/vm{1..3}
[root@slave2 nfs-sc]# cd /data/nfs-sc/
[root@slave2 nfs-sc]# ls
index.html  vm1  vm2  vm3
systemctl restart nfs rpcbind
[root@slave2 nfs-sc]# showmount -e slave2
Export list for slave2:
/data/nfs-sc/vm3 *
/data/nfs-sc/vm2 *
/data/nfs-sc/vm1 *
/data/nfs-sc     (everyone)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: vm1 #自定义pv名字
spec:
  capacity:
    storage: 100M #定义这个pv的限制存储大小
  accessModes:
    - ReadWriteMany #定义操作的权限
  storageClassName: nfs #自定义定义存储的类名,特定类的PV只能绑定到请求该类的PVC。没有storageClassName的PV没有类，只能绑定到不请求特定类的PVC
  nfs:
    path: /data/nfs-sc/vm1 #绑定主机的的路径
    server: 192.168.217.18 #指定nfs主机的ip地址
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vm2 #自定义pv名字
spec:
  capacity:
    storage: 1Gi #定义这个pv的限制存储大小
  accessModes:
    - ReadWriteMany #定义操作的权限
  storageClassName: nfs #自定义定义存储的类名,特定类的PV只能绑定到请求该类的PVC。没有storageClassName的PV没有类，只能绑定到不请求特定类的PVC
  nfs:
    path: /data/nfs-sc/vm2 #绑定主机的的路径
    server: 192.168.217.18 #指定nfs主机的ip地址
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vm3 #自定义pv名字
spec:
  capacity:
    storage: 3Gi #定义这个pv的限制存储大小
  accessModes:
    - ReadWriteMany #定义操作的权限
  storageClassName: nfs #自定义定义存储的类名,特定类的PV只能绑定到请求该类的PVC。没有storageClassName的PV没有类，只能绑定到不请求特定类的PVC
  nfs:
    path: /data/nfs-sc/vm3 #绑定主机的的路径
    server: 192.168.217.18 #指定nfs主机的ip地址

[root@master nginx]# k apply -f test-pv.yaml 
persistentvolume/vm1 unchanged
persistentvolume/vm2 created
persistentvolume/vm3 created
[root@master nginx]# k get pv
NAME       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM           STORAGECLASS   REASON   AGE
nginx-pv   3Gi        RWO            Retain           Bound       web/nginx-pvc                           3h39m
vm1        100M        RWX            Retain           Available                   nfs                     32s
vm2        1Gi        RWX            Retain           Available                   nfs                     10s
vm3        3Gi        RWX            Retain           Available                   nfs                     10s

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-pvc #这个名字随便了，一哈pod部署的时候会引用这个名字
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2G #定义要申请的空间大小
  storageClassName: nfs #这里要和我们定义pv那里storageClassName一样才行

[root@master nginx]# k get pv,pvc -A
NAME                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM               STORAGECLASS   REASON   AGE
persistentvolume/nginx-pv   3Gi        RWO            Retain           Bound       web/nginx-pvc                               3h58m
persistentvolume/vm1        100M       RWX            Retain           Available                       nfs                     19m
persistentvolume/vm2        1Gi        RWX            Retain           Available                       nfs                     19m
persistentvolume/vm3        3Gi        RWX            Retain           Bound       default/nginx-pvc   nfs                     19m
NAMESPACE   NAME                              STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
default     persistentvolumeclaim/nginx-pvc   Bound    vm3        3Gi        RWX            nfs            10m

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx-deploy-pvc
  name: nginx-deploy-pvc
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-deploy-pvc
  template:
    metadata:
      labels:
        app: nginx-deploy-pvc
    spec:
      containers:
      - image: nginx
        name: nginx
        volumeMounts: #这里定义pod中要挂载的路径
        - name: html
          mountPath: /usr/share/nginx/html
      volumes:
        - name: html #和上面的挂载目录一致
          persistentVolumeClaim:
            claimName: nginx-pvc #这里要绑定我们创建的pvc

[root@slave2 vm3]# pwd
/data/nfs-sc/vm3
[root@slave2 vm3]# echo "test pv and pvc" >index.html

[root@master nginx]# k exec -it nginx-deploy-pvc-98bdcc6cd-25x4w -- /bin/bash
root@nginx-deploy-pvc-98bdcc6cd-25x4w:/# curl localhost 
test pv and pvc

[root@master nfs-sc]# cat test-claim.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
  annotations: 
    volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi

[root@master nfs-sc]# cat test-claim.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi
  volumeMode: Filesystem
  storageClassName: managed-nfs-storage