阿里云K8S服务支持CSI存储卷-阿里云开发者社区

阿里云K8S服务支持CSI存储卷

2018-08-07 9894

版权

本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。

本文涉及的产品

容器镜像服务 ACR，镜像仓库100个不限时长

简介： 目前阿里云CSI存储插件支持云盘、OSS、NAS等存储类型，并支持通过StorageClass创建动态存储卷；开源地址：https://github.com/AliyunContainerService/csi-plugin

CSI插件简介

CSI是Container Storage Interface的简称，为容器编排系统和存储系统之间建立一套标准的存储调用接口。

Kubernetes支持CSI接口标准，并在1.10版本进入beta阶段。K8S除了支持CSI方式提供存储接口外，还有In-Tree方式、Flexvolume方式提供的各种存储接口。

In-Tree方式是嵌入在K8S源码内部的存储挂载实现，存在以下问题：

存储插件需要一同随K8S发布。
存储插件的问题有可能会影响K8S部件正常运行。
存储插件享有K8S部件同等的特权存在安全隐患。
存储插件开发者必须遵循K8S社区的规则开发代码。

FlexVolume机制通过调用一个可执行文件方式去实现挂载，它能够做到让存储提供方进行独立开发维护。

部署可执行文件时，需要host的root权限，依然存在安全隐患。
存储插件在执行mount、attach操作时，往往需要到host去安装第三方工具或加载依赖库，这样会使部署变得复杂。

CSI标准的出现解决了上述问题，后续K8S与存储系统的集成标准将全面支持CSI。

阿里云存储支持CSI Plugin

目前阿里云CSI存储插件支持云盘、OSS、NAS等存储类型，并支持通过StorageClass创建动态存储卷；

开源地址：https://github.com/AliyunContainerService/csi-plugin

插件的部署拓扑如下：

使用演示

云盘CSI Plugin部署

环境准备

通过阿里云容器服务创建K8S集群（1.10版本及以上），或者自己通过阿里云ECS服务器搭建K8S集群。
配置Kubelet启动参数：--enable-controller-attach-detach=true；

部署 CSI Attacher

# kubectl create -f attacher.yaml

CSI attacher使用0.2.0版本，attacher.yaml内容如下：

# This YAML file contains RBAC API objects,
# which are necessary to run external csi attacher for cinder.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-attacher

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-attacher-runner
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-attacher-role
subjects:
  - kind: ServiceAccount
    name: csi-attacher
    namespace: default
roleRef:
  kind: ClusterRole
  name: external-attacher-runner
  apiGroup: rbac.authorization.k8s.io
---

kind: Service
apiVersion: v1
metadata:
  name: csi-attacher
  labels:
    app: csi-attacher
spec:
  selector:
    app: csi-attacher
  ports:
    - name: dummy
      port: 12345

---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: csi-attacher
spec:
  serviceName: "csi-attacher"
  replicas: 1
  template:
    metadata:
      labels:
        app: csi-attacher
    spec:
      tolerations:
      - effect: NoSchedule
        operator: Exists
        key: node-role.kubernetes.io/master
      - effect: NoSchedule
        operator: Exists
        key: node.cloudprovider.kubernetes.io/uninitialized
      nodeSelector:
         node-role.kubernetes.io/master: ""
      serviceAccount: csi-attacher
      containers:
        - name: csi-attacher
          image: registry.cn-hangzhou.aliyuncs.com/plugins/csi-attacher:v0.2.0
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
          env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-diskplugin/csi.sock
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/csi-diskplugin
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-diskplugin
            type: DirectoryOrCreate

部署CSI Provsioner

# kubectl create -f provisioner.yaml

CSI provisioner使用0.2.0版本，内容如下：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-provisioner

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-provisioner-role
subjects:
  - kind: ServiceAccount
    name: csi-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: external-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

---
kind: Service
apiVersion: v1
metadata:
  name: csi-provisioner
  labels:
    app: csi-provisioner
spec:
  selector:
    app: csi-provisioner
  ports:
    - name: dummy
      port: 12345

---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: csi-provisioner
spec:
  serviceName: "csi-provisioner"
  replicas: 1
  template:
    metadata:
      labels:
        app: csi-provisioner
    spec:
      tolerations:
      - effect: NoSchedule
        operator: Exists
        key: node-role.kubernetes.io/master
      - effect: NoSchedule
        operator: Exists
        key: node.cloudprovider.kubernetes.io/uninitialized
      nodeSelector:
         node-role.kubernetes.io/master: ""
      serviceAccount: csi-provisioner
      containers:
        - name: csi-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/plugins/csi-provisioner:v0.2.0
          args:
            - "--provisioner=csi-diskplugin"
            - "--csi-address=$(ADDRESS)"
            - "--v=5"
          env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-diskplugin/csi.sock
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/csi-diskplugin
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-diskplugin
            type: DirectoryOrCreate

部署Disk Plugin

Disk Plugin部署为DaemonSet应用，命令：

# kubectl create -f plugin.yaml

CSI driver-registrar使用0.2.0版本；

Socket地址设置为：/var/lib/kubelet/plugins/csi-diskplugin/csi.sock；

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-diskplugin

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-diskplugin
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "update"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-diskplugin
subjects:
  - kind: ServiceAccount
    name: csi-diskplugin
    namespace: default
roleRef:
  kind: ClusterRole
  name: csi-diskplugin
  apiGroup: rbac.authorization.k8s.io

---
# This YAML file contains driver-registrar & csi driver nodeplugin API objects,
# which are necessary to run csi nodeplugin for disk.

kind: DaemonSet
apiVersion: apps/v1beta2
metadata:
  name: csi-diskplugin
spec:
  selector:
    matchLabels:
      app: csi-diskplugin
  template:
    metadata:
      labels:
        app: csi-diskplugin
    spec:
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      serviceAccount: csi-diskplugin
      hostNetwork: true
      hostPID: true
      containers:
        - name: driver-registrar
          image: registry.cn-hangzhou.aliyuncs.com/plugins/driver-registrar:v0.2.0
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
          env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-diskplugin/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/csi-diskplugin
        - name: csi-diskplugin
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
          image: registry.cn-hangzhou.aliyuncs.com/plugins/csi-diskplugin:v1.10-7424d08
          args :
            - "--nodeid=$(NODE_ID)"
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--v=5"
          env:
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: unix://var/lib/kubelet/plugins/csi-diskplugin/csi.sock
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: plugin-dir
              mountPath: /var/lib/kubelet/plugins/csi-diskplugin
            - name: pods-mount-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: "Bidirectional"
            - mountPath: /dev
              name: host-dev
            - mountPath: /sys
              name: host-sys
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /run/dbus
              name: host-run
              readOnly: true
            - mountPath: /var/log/alicloud
              name: host-log
      volumes:
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-diskplugin
            type: DirectoryOrCreate
        - name: pods-mount-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: Directory
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-diskplugin
            type: DirectoryOrCreate
        - name: host-dev
          hostPath:
            path: /dev
        - name: host-run
          hostPath:
            path: /run/dbus
        - name: host-sys
          hostPath:
            path: /sys
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: host-log
          hostPath:
            path: /var/log/alicloud/

部署检查

# kubectl get pod

NAME                                 READY     STATUS    RESTARTS   AGE
csi-attacher-0                       1/1       Running   0          1d
csi-diskplugin-5w7wz                 2/2       Running   0          1d
csi-diskplugin-j99h9                 2/2       Running   0          1d
csi-diskplugin-ljg9s                 2/2       Running   0          1d
csi-diskplugin-nhbfp                 2/2       Running   0          1d
csi-provisioner-0                    1/1       Running   0          1d

使用CSI插件挂载云盘

下面以创建动态云盘数据卷为例，介绍如何使用CSI插件挂载；

创建云盘StorageClass

定义一个SSD、ReadWrite、北京B区的StorageClass；

# kubectl create -f storageclass.yaml

storageclass.yaml内容如下：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-disk
provisioner: csi-diskplugin
parameters:
    zoneId: cn-beijing-b
    regionId: cn-beijing
    fsType: ext4
    type: cloud_ssd
    readOnly: "false"
reclaimPolicy: Delete

创建应用

创建Nginx服务并通过PVC挂载云盘，在PVC配置csi-disk storageClassName；

# kubectl create -f nginx.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: disk-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 25Gi
  storageClassName: csi-disk
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment1
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        volumeMounts:
          - name: disk-pvc
            mountPath: "/data"
      volumes:
        - name: disk-pvc
          persistentVolumeClaim:
            claimName: disk-pvc

云盘挂载验证

查看nginx容器：

# kubectl get pod | grep nginx
# nginx-deployment1-5879d9db88-wr9j9   1/1       Running   0          1d

查看容器内挂载信息：

# kubectl exec -ti nginx-deployment1-5879d9db88-wr9j9 df | grep data
/dev/vdd        25671908   45084  24299720   1% /data
# kubectl exec -ti nginx-deployment1-5879d9db88-wr9j9 touch /data/aliyun
# kubectl exec -ti nginx-deployment1-5879d9db88-wr9j9 ls /data
aliyun    lost+found

删除容器，验证云盘高可用：

# kubectl delete pod nginx-deployment1-5879d9db88-wr9j9
# kubectl get pod
NAME                                 READY     STATUS    RESTARTS   AGE
nginx-deployment1-5879d9db88-mjb5p   1/1       Running   0          14s
# kubectl exec -ti nginx-deployment1-5879d9db88-mjb5p ls /data
aliyun    lost+found

上述操作通过CSI插件成功挂载阿里云云盘，并验证了应用数据的高可用性；

阿里云K8S服务支持CSI存储卷

CSI插件简介

阿里云存储支持CSI Plugin

使用演示

云盘CSI Plugin部署

环境准备

部署 CSI Attacher

部署CSI Provsioner

部署Disk Plugin

部署检查

使用CSI插件挂载云盘

创建云盘StorageClass

创建应用

云盘挂载验证

阿里云容器服务 ACK

热门文章

最新文章

相关产品

相关课程

相关电子书

相关实验场景

推荐镜像