如何在Kubernetes上扩展MongoDB？

我们可以把StatefulSets的特性归纳如下：

1. 有序索引Pod
2. 稳定的网络ID
3. 有序并行的Pod管理
4. 滚动更新

1. 配置HostVM的Daemon Set
2. Mongo Pod的Service Account和ClusterRole Binding
3. 为Pod提供永久性存储SSDs的Storage Class
4. 访问Mongo容器的Headless Service
5. Mongo Pods Stateful Set
6. GCP Internal LB：从Kubernetes集群外部访问MongoDB（可选）
7. 使用Ingress访问Pod（可选）

Daemon Set for HostVM Configuration

kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
name: hostvm-configurer
labels:
app: startup-script
spec:
template:
metadata:
 labels:
 app: startup-script
spec:
 hostPID: true
 containers:
 - name: hostvm-configurer-container
 image: gcr.io/google-containers/startup-script:v1
 securityContext:
 privileged: true
 env:
 - name: STARTUP_SCRIPT
 value: |
 #! /bin/bash
 set -o errexit
 set -o pipefail
 set -o nounset

 # Disable hugepages
 echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled
 echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag

Configuration for ServiceAccount, Storage Class, Headless SVC and StatefulSet

apiVersion: v1
kind: Namespace
metadata:
name: mongo
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: mongo
namespace: mongo
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: mongo
subjects:
- kind: ServiceAccount
name: mongo
namespace: mongo
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
---
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: fast
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-ssd
fsType: xfs
allowVolumeExpansion: true
---
apiVersion: v1
kind: Service
metadata:
name: mongo
namespace: mongo
labels:
name: mongo
spec:
ports:
- port: 27017
targetPort: 27017
clusterIP: None
selector:
role: mongo
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: mongo
namespace: mongo
spec:
serviceName: mongo
replicas: 3
template:
metadata:
 labels:
 role: mongo
 environment: staging
 replicaset: MainRepSet
spec:
 affinity:
 podAntiAffinity:
 preferredDuringSchedulingIgnoredDuringExecution:
 - weight: 100
 podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: replicaset
 operator: In
 values:
 - MainRepSet
 topologyKey: kubernetes.io/hostname
 terminationGracePeriodSeconds: 10
 serviceAccountName: mongo
 containers:
 - name: mongo
 image: mongo
 command:
 - mongod
 - "--wiredTigerCacheSizeGB"
 - "0.25"
 - "--bind_ip"
 - "0.0.0.0"
 - "--replSet"
 - MainRepSet
 - "--smallfiles"
 - "--noprealloc"
 ports:
 - containerPort: 27017
 volumeMounts:
 - name: mongo-persistent-storage
 mountPath: /data/db
 resources:
 requests:
 cpu: 1
 memory: 2Gi
 - name: mongo-sidecar
 image: cvallance/mongo-k8s-sidecar
 env:
 - name: MONGO_SIDECAR_POD_LABELS
 value: "role=mongo,environment=staging"
 - name: KUBE_NAMESPACE
 value: "mongo"
 - name: KUBERNETES_MONGO_SERVICE_NAME
 value: "mongo"
volumeClaimTemplates:
- metadata:
 name: mongo-persistent-storage
 annotations:
 volume.beta.kubernetes.io/storage-class: "fast"
spec:
 accessModes: [ "ReadWriteOnce" ]
 storageClassName: fast
 resources:
 requests:
 storage: 10Gi

1. 应该使用适当的环境变量仔细配置Mongo的Sidecar，以及为Pod提供的标签，和为deployment和service的命名空间。 有关Sidecar容器的详细信息，请点击此处。
2. 默认缓存大小的指导值是：“50％的RAM减去1GB，或256MB”。 鉴于所请求的内存量为2GB，此处的WiredTiger缓存大小已设置为256MB。
3. Inter-Pod Anti-Affinity确保在同一个工作节点上不会安排2个Mongo Pod，从而使其能够适应节点故障。 此外，建议将节点保留在不同的可用区中，以便集群能够抵御区域故障。
4. 当前部署的Service Account具有管理员权限。 但是，它应该仅限于DB的命名空间。

部署MongoDB集群

kubectl apply -f configure-node.yml
kubectl apply -f mongo.yml

root$ kubectl -n mongo get all
NAME DESIRED CURRENT AGE
statefulsets/mongo 3 3 3m
NAME READY STATUS RESTARTS AGE
po/mongo-0 2/2 Running 0 3m
po/mongo-1 2/2 Running 0 2m
po/mongo-2 2/2 Running 0 1m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/mongo ClusterIP None <none> 27017/TCP 3m

kubectl run my-shell --rm -i --tty --image ubuntu -- bash
root@my-shell-68974bb7f7-cs4l9:/# dig mongo.mongo +search +noall +answer
; <<>> DiG 9.11.3-1ubuntu1.1-Ubuntu <<>> mongo.mongo +search +noall +answer
;; global options: +cmd
mongo.mongo.svc.cluster.local. 30 IN A 10.56.7.10
mongo.mongo.svc.cluster.local. 30 IN A 10.56.8.11
mongo.mongo.svc.cluster.local. 30 IN A 10.56.1.4

root@my-shell-68974bb7f7-cs4l9:/# nslookup 10.56.6.17
17.6.56.10.in-addr.arpa name = mongo-0.mongo.mongo.svc.cluster.local.
root@my-shell-68974bb7f7-cs4l9:/# nslookup 10.56.7.10
10.7.56.10.in-addr.arpa name = mongo-1.mongo.mongo.svc.cluster.local.
root@my-shell-68974bb7f7-cs4l9:/# nslookup 10.56.8.11
11.8.56.10.in-addr.arpa name = mongo-2.mongo.mongo.svc.cluster.local.

Node-0: mongo-0.mongo.mongo.svc.cluster.local:27017 
Node-1: mongo-1.mongo.mongo.svc.cluster.local:27017 
Node-2: mongo-2.mongo.mongo.svc.cluster.local:27017

GCP Internal LB SVC Configuration（可选）

apiVersion: v1
kind: Service
metadata: 
annotations: 
cloud.google.com/load-balancer-type: Internal
name: mongo-0
namespace: mongo
spec: 
ports: 
- 
 port: 27017
 targetPort: 27017
selector: 
statefulset.kubernetes.io/pod-name: mongo-0
type: LoadBalancer

root$ kubectl -n mongo get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mongo ClusterIP None <none> 27017/TCP 15m
mongo-0 LoadBalancer 10.59.252.157 10.20.20.2 27017:30184/TCP 9m
mongo-1 LoadBalancer 10.59.252.235 10.20.20.3 27017:30343/TCP 9m
mongo-2 LoadBalancer 10.59.254.199 10.20.20.4 27017:31298/TCP 9m

通过Ingress访问Pod（可选）

...
spec:
rules:
- host: mongo.example.com
http:
 paths:
 - path: '/mongo-0'
 backend:
 hostNames:
 - mongo-0
 serviceName: mongo # There is no extra service. This is the headless service.
 servicePort: '27017' 

root$ kubectl -n mongo scale statefulsets mongo --replicas=4
statefulset "mongo" scaled
root$ kubectl -n mongo get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
mongo-0 2/2 Running 0 25m 10.56.6.17 gke-k8-demo-demo-k8-pool-1-45712bb7-vfqs
mongo-1 2/2 Running 0 24m 10.56.7.10 gke-k8-demo-demo-k8-pool-1-c6901f2e-trv5
mongo-2 2/2 Running 0 23m 10.56.8.11 gke-k8-demo-demo-k8-pool-1-c7622fba-qayt
mongo-3 2/2 Running 0 3m 10.56.1.4 gke-k8-demo-demo-k8-pool-1-85308bb7-89a4

root$ kubectl -n mongo get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mongo-persistent-storage-mongo-0 Bound pvc-337fb7d6-9f8f-11e8-bcd6-42010a940024 11G RWO fast 49m
mongo-persistent-storage-mongo-1 Bound pvc-53375e31-9f8f-11e8-bcd6-42010a940024 11G RWO fast 49m
mongo-persistent-storage-mongo-2 Bound pvc-6cee0f97-9f8f-11e8-bcd6-42010a940024 11G RWO fast 48m
mongo-persistent-storage-mongo-3 Bound pvc-3e89573f-9f92-11e8-bcd6-42010a940024 11G RWO fast 28m

进一步的考虑

1. 给Mongo Pod的Node Pool打上合适的label并确保在StatefulSets和HostVM配置的DaemonSets的Spec中指定适当的Node Affinity会很有帮助。 这是因为DaemonSet将调整主机操作系统的一些参数，并且这些设置应仅限于MongoDB Pod。 没有这些设置，对其他应用程序可能会更好。
2. 在GKE中给Node Pool打Label非常容易，可以直接从GCP控制台进行。
3. 虽然我们在Pod的Spec中指定了CPU和内存限制，但我们也可以考虑部署VPA（Vertical Pod Autoscaler）。
4. 可以通过实施网络策略或服务网格（如Istio）来控制从集群内部到我们的数据库的流量。

如果你已经看到这里，我相信你已经浏览了整个博文。 我试图整理很多分散的信息并将其作为一个整体呈现。 我的目标是为您提供足够的信息，以便开始使用Kubernetes上的Stateful Sets，并希望你们中的许多人觉得它很有用。 我们非常欢迎您提出反馈、意见或建议。:)

本文转自DockOne-如何在Kubernetes上扩展MongoDB？