Kubernetes环境中使用NAS动态存储卷-阿里云开发者社区

开发者社区> 阿里云容器服务 ACK> 正文

Kubernetes环境中使用NAS动态存储卷

简介: 介绍 K8S在应用中经常会用到nas共享存储卷,通常的使用方法是通过一个pv、pvc进行挂载,但当需要pv、pvc数量很大的时候,手动创建显得非常繁琐,这时动态数据卷的功能可以满足您的需求。 以下场景: 当为不同的用户提供不同的nas子目录进行挂载的时候,可以考虑使用nas动态存储卷; 当使用S.

介绍

K8S在应用中经常会用到nas共享存储卷,通常的使用方法是通过一个pv、pvc进行挂载,但当需要pv、pvc数量很大的时候,手动创建显得非常繁琐,这时动态数据卷的功能可以满足您的需求。

以下场景:

当为不同的用户提供不同的nas子目录进行挂载的时候,可以考虑使用nas动态存储卷;

当使用SatefulSet部署应用,每个pod配置使用不同nas数据卷的时候;

本方案中实现的动态nas卷,是在某个nas文件系统下通过创建子目录、并把子目录映射为一个动态pv提供给应用;

部署详解

1. 部署Controller

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: alicloud-nas-controller
  namespace: kube-system
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: alicloud-nas-controller
    spec:
      tolerations:
      - operator: "Exists"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: Exists
      priorityClassName: system-node-critical
      serviceAccount: admin
      hostNetwork: true
      containers:
        - name: nfs-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/acs/alicloud-nas-controller:v1.14.3.8-58bf821-aliyun
          env:
          - name: PROVISIONER_NAME
            value: alicloud/nas
          securityContext:
            privileged: true
          volumeMounts:
          - mountPath: /var/log
            name: log
      volumes:
      - hostPath:
          path: /var/log
        name: log

2. StorageClass配置:

Nas Controller支持两种挂载驱动,flexvolume、nfs方式;

flexvolume:表示使用阿里云提供的flexvolume(alicloud/nas)作为挂载驱动,集群中需要部署flexvolume;

nfs:表示用时Kubernetes提供的标准挂载驱动;

Flexvolume方式:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-nas
mountOptions:
- nolock,tcp,noresvport
- vers=3
parameters:
  server: "23a9649583-iaq37.cn-shenzhen.nas.aliyuncs.com:/nasroot1/,23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com:/nasroot2/"
  driver: flexvolume
provisioner: alicloud/nas
reclaimPolicy: Delete

mountOptions:

表示生成的pv option 配置,挂载nas卷时使用这个option挂载;

server:

表示生成目标pv所使用nas挂载点列表;格式为 nfsurl1:/path1,nfsurl2:/path2;

当配置多个server的时候,通过此storageclass创建的pv会轮询使用上述server作为配置参数;

driver:

支持flexvolume、nfs,默认为nfs;

reclaimPolicy:

pv的回收策略,建议配置为Retain;当配置为Delete时,删除pv后nas文件系统中的对应目录会默认修改名字(path-name ==> archived-path-name);

如果需要删除文件系统中对应的存储目录,可在storageclass中配置archiveOnDelete为false;

NFS 方式:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-nas
mountOptions:
- noresvport
- vers=4.0
parameters:
  server: "23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com:/nasroot2/"
  driver: nfs
provisioner: alicloud/nas
reclaimPolicy: Delete

应用示例

1. 创建Flexvolume驱动数据卷

创建存储类:# kubectl create -f -

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-nas
mountOptions:
- nolock,tcp,noresvport
- vers=3
parameters:
  server: "23a9649583-iaq37.cn-shenzhen.nas.aliyuncs.com:/nasroot1/,23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com:/nasroot2/"
  driver: flexvolume
provisioner: alicloud/nas
reclaimPolicy: Delete

创建statefulset应用,配置动态存储卷:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 5
  volumeClaimTemplates:
  - metadata:
      name: html
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: alicloud-nas
      resources:
        requests:
          storage: 2Gi
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        volumeMounts:
        - mountPath: "/data"
          name: html

检查生成的pvc、pv;

# kubectl get pvc | grep web
html-web-0      Bound         default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        96s
html-web-1      Bound         default-html-web-1-pvc-6b108a5b-b9e0-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        8m14s
html-web-2      Bound         default-html-web-2-pvc-661ebd73-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        73s
html-web-3      Bound         default-html-web-3-pvc-69569748-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        68s
html-web-4      Bound         default-html-web-4-pvc-6c52a277-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        63s

下面看到使用了flexvolume驱动创建的pv:
# kubectl get pv default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf -oyaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: alicloud/nas
  creationTimestamp: "2019-08-08T13:35:20Z"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    createdby.aliyun.com: alicloud-nas-controller
    version.controller.aliyun.com: v1.14.3-1173ac8
  name: default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf
  resourceVersion: "2241971"
  selfLink: /api/v1/persistentvolumes/default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf
  uid: 5da7280c-b9e1-11e9-91a9-00163e10b7cf
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: html-web-0
    namespace: default
    resourceVersion: "2241962"
    uid: 58450866-b9e1-11e9-91a9-00163e10b7cf
  flexVolume:
    driver: alicloud/nas
    options:
      options: nolock,tcp,noresvport,vers=3
      path: /nasroot2/default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf
      server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
      vers: "3"
  persistentVolumeReclaimPolicy: Delete
  storageClassName: alicloud-nas
  volumeMode: Filesystem
status:
  phase: Bound

检查5个pv是否轮询使用server中的不同挂载点:

# for item in `kubectl get pv | grep web | awk '{print $1}'`; do  kubectl get pv $item -oyaml | grep server; done
      server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
      server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
      server: 23a9649583-iaq37.cn-shenzhen.nas.aliyuncs.com
      server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
      server: 23a9649583-iaq37.cn-shenzhen.nas.aliyuncs.com
      
# for item in `kubectl get pv | grep web | awk '{print $1}'`; do  kubectl get pv $item -oyaml | grep path; done
      path: /nasroot2/default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf
      path: /nasroot2/default-html-web-1-pvc-6b108a5b-b9e0-11e9-91a9-00163e10b7cf
      path: /nasroot1/default-html-web-2-pvc-661ebd73-b9e1-11e9-91a9-00163e10b7cf
      path: /nasroot2/default-html-web-3-pvc-69569748-b9e1-11e9-91a9-00163e10b7cf
      path: /nasroot1/default-html-web-4-pvc-6c52a277-b9e1-11e9-91a9-00163e10b7cf

nas文件系统中生成不同的子目录:

# ls nasroot1
default-html-web-2-pvc-661ebd73-b9e1-11e9-91a9-00163e10b7cf  default-html-web-4-pvc-6c52a277-b9e1-11e9-91a9-00163e10b7cf
# ls nasroot2
default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf  default-html-web-3-pvc-69569748-b9e1-11e9-91a9-00163e10b7cf
default-html-web-1-pvc-6b108a5b-b9e0-11e9-91a9-00163e10b7cf

删除应用pv、pvc,nas文件系统中目录名字改为archived-**

# ls nasroot1
archived-default-html-web-2-pvc-661ebd73-b9e1-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37
archived-default-html-web-4-pvc-6c52a277-b9e1-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37
# ls nasroot2
archived-default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37
archived-default-html-web-1-pvc-6b108a5b-b9e0-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37
archived-default-html-web-3-pvc-69569748-b9e1-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37

2. 创建NFS驱动数据卷

创建存储类:# kubectl create -f -

archiveOnDelete: 配置为false,表示pv回收策略为Delete时,删除pv会把文件系统中的目录一起删除;

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-nas
mountOptions:
- noresvport
- vers=4.0
parameters:
  server: "23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com:/nasroot2/"
  driver: nfs
  archiveOnDelete: "false"
provisioner: alicloud/nas
reclaimPolicy: Delete

创建statefulset,检查pv、pvc:

# kubectl get pvc | grep web
html-web-0      Bound         default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        2m4s
html-web-1      Bound         default-html-web-1-pvc-015441f5-b9e2-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        2m
html-web-2      Bound         default-html-web-2-pvc-3edcb47b-b9e2-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        17s
html-web-3      Bound         default-html-web-3-pvc-420f9bd8-b9e2-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        11s
html-web-4      Bound         default-html-web-4-pvc-4843579f-b9e2-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        1s

下面信息显示,pv使用的驱动为NFS:
# kubectl get pv default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf -oyaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: alicloud/nas
  creationTimestamp: "2019-08-08T13:39:50Z"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    createdby.aliyun.com: alicloud-nas-controller
    version.controller.aliyun.com: v1.14.3-1173ac8
  name: default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf
  resourceVersion: "2243033"
  selfLink: /api/v1/persistentvolumes/default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf
  uid: fef7e4d1-b9e1-11e9-91a9-00163e10b7cf
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: html-web-0
    namespace: default
    resourceVersion: "2243017"
    uid: fed53a89-b9e1-11e9-91a9-00163e10b7cf
  mountOptions:
  - noresvport
  - vers=4.0
  nfs:
    path: /nasroot2/default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
  persistentVolumeReclaimPolicy: Delete
  storageClassName: alicloud-nas
  volumeMode: Filesystem
status:
  phase: Bound

检查5个pv的挂载点:

# for item in `kubectl get pv | grep web | awk '{print $1}'`; do  kubectl get pv $item -oyaml | grep server; done
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
# for item in `kubectl get pv | grep web | awk '{print $1}'`; do  kubectl get pv $item -oyaml | grep path; done
    path: /nasroot2/default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf
    path: /nasroot2/default-html-web-1-pvc-015441f5-b9e2-11e9-91a9-00163e10b7cf
    path: /nasroot2/default-html-web-2-pvc-3edcb47b-b9e2-11e9-91a9-00163e10b7cf
    path: /nasroot2/default-html-web-3-pvc-420f9bd8-b9e2-11e9-91a9-00163e10b7cf
    path: /nasroot2/default-html-web-4-pvc-4843579f-b9e2-11e9-91a9-00163e10b7cf

nas文件系统中生成不同的子目录:

# ls nasroot2/
default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf  default-html-web-3-pvc-420f9bd8-b9e2-11e9-91a9-00163e10b7cf
default-html-web-1-pvc-015441f5-b9e2-11e9-91a9-00163e10b7cf  default-html-web-4-pvc-4843579f-b9e2-11e9-91a9-00163e10b7cf
default-html-web-2-pvc-3edcb47b-b9e2-11e9-91a9-00163e10b7cf

删除应用pv、pvc,nas文件系统中目录同时被删除;(StorageClass中配置了archiveOnDelete: "false")

版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

分享:
阿里云容器服务 ACK
使用钉钉扫一扫加入圈子
+ 订阅

云端最佳容器应用运行环境,安全、稳定、极致弹性

官方博客
官网链接