K8S有状态服务-动态云盘使用最佳实践

介绍

动态挂载方式是指在应用中显式声明PVC，并在PVC中声明StorageClass；这时应用会通过Storageclass中指定的Provisioner来自动创建云盘，并自动生成云盘PV资源类型；

使用动态云盘需要满足以下条件：

集群中要部署云盘Provisioner服务，实现自动创建云盘；
创建预期使用storageclass资源，并指定云盘Provisioner；
在PVC中显式声明使用哪个storageclass；

无需显式创建PV，而是通过Provisioner自动创建；
无需在ecs控制台购买云盘，在应用部署时自动购买的情况；

云盘Provisioner

使用云盘动态卷的一个前提是系统中已经部署了云盘Provisioner。

K8S集群会默认部署Provisioner，Provisioner创建云盘需要对云盘有操作权限，可以通过AK、或STS token来获取权限；

配置AK：在部署Provisioner的时候设置ACCESS_KEY_ID、ACCESS_KEY_SECRET环境变量，可以配置ak；
配置STS：为默认方式，可以给集群（Master节点）授予RAM权限，详情参看RAM权限管理；

下面yaml文件为部署Provisioner的详细描述：

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: alicloud-disk-common
provisioner: alicloud/disk
parameters:
  type: cloud

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: alicloud-disk-efficiency
provisioner: alicloud/disk
parameters:
  type: cloud_efficiency

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: alicloud-disk-ssd
provisioner: alicloud/disk
parameters:
  type: cloud_ssd

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: alicloud-disk-available
provisioner: alicloud/disk
parameters:
  type: available

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: alicloud-disk-controller-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: alicloud-disk-controller
  namespace: kube-system

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: run-alicloud-disk-controller
subjects:
  - kind: ServiceAccount
    name: alicloud-disk-controller
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: alicloud-disk-controller-runner
  apiGroup: rbac.authorization.k8s.io

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: alicloud-disk-controller
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: alicloud-disk-controller
    spec:
      tolerations:
      - effect: NoSchedule
        operator: Exists
        key: node-role.kubernetes.io/master
      - effect: NoSchedule
        operator: Exists
        key: node.cloudprovider.kubernetes.io/uninitialized
      nodeSelector:
         node-role.kubernetes.io/master: ""
      serviceAccount: alicloud-disk-controller
      containers:
        - name: alicloud-disk-controller
          image: registry.cn-hangzhou.aliyuncs.com/acs/alicloud-disk-controller:v1.10.4-f431fd8
          volumeMounts:
            - name: cloud-config
              mountPath: /etc/kubernetes/
            - name: logdir
              mountPath: /var/log/alicloud/
      volumes:
        - name: cloud-config
          hostPath:
            path: /etc/kubernetes/
        - name: logdir
          hostPath:
            path: /var/log/alicloud/

StorageClass

阿里云K8S系统初始化的时候会默认创建4个StorageClass，这4个StorageClass适合在集群类型为单一zone的情况，若为多zone部署的集群，则需要自己另行创建；

alicloud-disk-common：创建普通云盘。
alicloud-disk-efficiency：创建高效云盘。
alicloud-disk-ssd：创建SSD云盘。
alicloud-disk-available：提供高可用选项，先试图创建高效云盘；如果相应AZ的高效云盘资源售尽，再试图创建SSD盘；如果SSD售尽，则试图创建普通云盘。

下面yaml描述了创建Storageclass的细节：

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: alicloud-disk-common-hangzhou-b
provisioner: alicloud/disk
reclaimPolicy: Retain
parameters:
  type: cloud_ssd
  regionid: cn-hangzhou
  zoneid: cn-hangzhou-b
  fstype: "ext4"
  readonly: "false"
  encrypted: "true"

reclaimPolicy：表示创建pv的回收策略，支持Delete、Retain两个类型，默认为Delete；这里需要注意：配置为Delete时，删除PVC后云盘一起被删除，数据不可恢复。

type: 表示创建什么类型的云盘，支持cloud、cloud_efficiency、cloud_ssd、available类型；

regionid：表示创建云盘所在region；

zoneid：表示创建云盘所在zone；

fstype：表示云盘使用的文件系统，可选项，默认为ext4；

readonly：表示挂载的读写权限是否为只读，可选项，默认为false；

encrypted：是否创建加密云盘，可选项，默认为false；

使用动态卷创建应用

部署一下应用模板，在PVC中显式指定storageClassName为上述创建的StorageClass；

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: disk-ssd
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: alicloud-disk-ssd-beijing-b
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-dynamic
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
          - name: disk-pvc
            mountPath: "/data"
      volumes:
        - name: disk-pvc
          persistentVolumeClaim:
            claimName: disk-ssd

验证高可用

创建应用

# kubectl create -f dynamic.yaml

# kubectl get pod | grep dynamic
nginx-dynamic-69f9bd7b8c-58sbs   1/1       Running   0          3m

# kubectl exec nginx-dynamic-69f9bd7b8c-58sbs df | grep data
/dev/vdb        20511312   45080  19401272   1% /data

在云盘中创建文件：

# kubectl exec nginx-dynamic-69f9bd7b8c-58sbs ls /data
lost+found

# kubectl exec nginx-dynamic-69f9bd7b8c-58sbs touch /data/dynamic
# kubectl exec nginx-dynamic-69f9bd7b8c-58sbs ls /data
dynamic
lost+found

删除Pod，验证文件持久化：

# kubectl delete pod nginx-dynamic-69f9bd7b8c-58sbs
pod "nginx-dynamic-69f9bd7b8c-58sbs" deleted

# kubectl get pod
NAME                             READY     STATUS              RESTARTS   AGE
nginx-dynamic-69f9bd7b8c-58sbs   0/1       Terminating         0          5m
nginx-dynamic-69f9bd7b8c-ddcbb   0/1       ContainerCreating   0          2s

# kubectl exec nginx-dynamic-69f9bd7b8c-ddcbb ls /data
dynamic
lost+found