6）最后，我们指定了每个 PersistentVolume 的大小为 10GB，我们可以根据自己的实际需要进行调整该值。最后，完整的elasticsaerch-statefulset.yaml资源清单文件内容如下：
cat elasticsaerch-statefulset.yaml
apiVersion: apps/v1 kind: StatefulSet metadata: name: es-cluster namespace: kube-logging spec: serviceName: elasticsearch replicas: 3 selector: matchLabels: app: elasticsearch template: metadata: labels: app: elasticsearch spec: containers: - name: elasticsearch image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0 imagePullPolicy: IfNotPresent resources: limits: cpu: 1000m requests: cpu: 100m ports: - containerPort: 9200 name: rest protocol: TCP - containerPort: 9300 name: inter-node protocol: TCP volumeMounts: - name: data mountPath: /usr/share/elasticsearch/data env: - name: cluster.name value: k8s-logs - name: node.name valueFrom: fieldRef: fieldPath: metadata.name - name: discovery.seed_hosts value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch" - name: cluster.initial_master_nodes value: "es-cluster-0,es-cluster-1,es-cluster-2" - name: ES_JAVA_OPTS value: "-Xms512m -Xmx512m" initContainers: - name: fix-permissions image: busybox imagePullPolicy: IfNotPresent command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"] securityContext: privileged: true volumeMounts: - name: data mountPath: /usr/share/elasticsearch/data - name: increase-vm-max-map image: busybox imagePullPolicy: IfNotPresent command: ["sysctl", "-w", "vm.max_map_count=262144"] securityContext: privileged: true - name: increase-fd-ulimit image: busybox imagePullPolicy: IfNotPresent command: ["sh", "-c", "ulimit -n 65536"] securityContext: privileged: true volumeClaimTemplates: - metadata: name: data labels: app: elasticsearch spec: accessModes: [ "ReadWriteOnce" ] storageClassName: do-block-storage resources: requests: storage: 10Gi apiVersion: apps/v1 kind: StatefulSet metadata: name: es-cluster namespace: kube-logging spec: serviceName: elasticsearch replicas: 3 selector: matchLabels: app: elasticsearch template: metadata: labels: app: elasticsearch spec: containers: - name: elasticsearch image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0 imagePullPolicy: IfNotPresent resources: limits: cpu: 1000m requests: cpu: 100m ports: - containerPort: 9200 name: rest protocol: TCP - containerPort: 9300 name: inter-node protocol: TCP volumeMounts: - name: data mountPath: /usr/share/elasticsearch/data env: - name: cluster.name value: k8s-logs - name: node.name valueFrom: fieldRef: fieldPath: metadata.name - name: discovery.seed_hosts value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch" - name: cluster.initial_master_nodes value: "es-cluster-0,es-cluster-1,es-cluster-2" - name: ES_JAVA_OPTS value: "-Xms512m -Xmx512m" initContainers: - name: fix-permissions image: busybox imagePullPolicy: IfNotPresent command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"] securityContext: privileged: true volumeMounts: - name: data mountPath: /usr/share/elasticsearch/data - name: increase-vm-max-map image: busybox imagePullPolicy: IfNotPresent command: ["sysctl", "-w", "vm.max_map_count=262144"] securityContext: privileged: true - name: increase-fd-ulimit image: busybox imagePullPolicy: IfNotPresent command: ["sh", "-c", "ulimit -n 65536"] securityContext: privileged: true volumeClaimTemplates: - metadata: name: data labels: app: elasticsearch spec: accessModes: [ "ReadWriteOnce" ] storageClassName: do-block-storage resources: requests: storage: 10Gi
kubectl apply -f elasticsaerch-statefulset.yaml
kubectl get pods -n kube-logging
显示如下，说明es创建成功了：
NAME READY STATUS RESTARTS AGE es-cluster-0 1/1 Running 0 2m8s es-cluster-1 1/1 Running 0 117s es-cluster-2 1/1 Running 0 107s
kubectl get svc -n kube-logging
显示如下
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 33m
pod部署完成之后，可以通过REST API检查elasticsearch集群是否部署成功，使用下面的命令将本地端口9200转发到 Elasticsearch 节点（如es-cluster-0）对应的端口：
kubectl port-forward es-cluster-0 9200:9200 --namespace=kube-logging
然后，在另外的终端窗口中，执行如下请求，新开一个master1终端：
curl http://localhost:9200/_cluster/state?pretty
输出如下：
{ "cluster_name" : "k8s-logs", "compressed_size_in_bytes" : 348, "cluster_uuid" : "QD06dK7CQgids-GQZooNVw", "version" : 3, "state_uuid" : "mjNIWXAzQVuxNNOQ7xR-qg", "master_node" : "IdM5B7cUQWqFgIHXBp0JDg", "blocks" : { }, "nodes" : { "u7DoTpMmSCixOoictzHItA" : { "name" : "es-cluster-1", "ephemeral_id" : "ZlBflnXKRMC4RvEACHIVdg", "transport_address" : "10.244.8.2:9300", "attributes" : { } }, "IdM5B7cUQWqFgIHXBp0JDg": { "name" : "es-cluster-0", "ephemeral_id" : "JTk1FDdFQuWbSFAtBxdxAQ", "transport_address" : "10.244.44.3:9300", "attributes" : { } }, "R8E7xcSUSbGbgrhAdyAKmQ" : { "name" : "es-cluster-2", "ephemeral_id" :"9wv6ke71Qqy9vk2LgJTqaA", "transport_address" : "10.244.40.4:9300", "attributes" : { } } }, ...
看到上面的信息就表明我们名为 k8s-logs的Elasticsearch 集群成功创建了3个节点：es-cluster-0，es-cluster-1，和es-cluster-2，当前主节点是 es-cluster-0。
#安装kibana组件
elasticsearch安装成功之后，开始部署kibana
cat kibana.yaml
apiVersion: v1 kind: Service metadata: name: kibana namespace: kube-logging labels: app: kibana spec: ports: - port: 5601 selector: app: kibana --- apiVersion: apps/v1 kind: Deployment metadata: name: kibana namespace: kube-logging labels: app: kibana spec: replicas: 1 selector: matchLabels: app: kibana template: metadata: labels: app: kibana spec: containers: - name: kibana image: docker.elastic.co/kibana/kibana:7.2.0 imagePullPolicy: IfNotPresent resources: limits: cpu: 1000m requests: cpu: 100m env: - name: ELASTICSEARCH_URL value: http://elasticsearch:9200 ports: - containerPort: 5601
上面我们定义了两个资源对象，一个Service和Deployment，为了测试方便，我们将 Service 设置为了 NodePort 类型，Kibana Pod 中配置都比较简单，唯一需要注意的是我们使用ELASTICSEARCH_URL 这个环境变量来设置Elasticsearch 集群的端点和端口，直接使用 Kubernetes DNS 即可，此端点对应服务名称为 elasticsearch，由于是一个 headless service，所以该域将解析为3个 Elasticsearch Pod 的 IP 地址列表。
配置完成后，直接使用 kubectl工具创建：
kubectl apply -f kibana.yaml
kubectl get pods -n kube-logging
显示如下，说明kibana也已经部署成功了
NAME READY STATUS RESTARTS AGE es-cluster-0 1/1 Running 0 170m es-cluster-1 1/1 Running 0 170m es-cluster-2 1/1 Running 0 170m kibana-5749b5778b-c9djr 1/1 Running 0 4m3s
kubectl get svc -n kube-logging
显示如下：
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 3h28m kibana ClusterIP 10.104.159.24 <none> 5601/TCP
修改service的type类型为NodePort：
kubectl edit svc kibana -n kube-logging
把type:ClusterIP变成type: NodePort
保存退出之后
kubectlget svc -n kube-logging
显示如下：
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE elasticsearchClusterIP None <none> 9200/TCP,9300/TCP 3h50m kibana NodePort 10.104.159.24 <none> 5601:32462/TCP 34m
在浏览器中打开http://<任意节点IP>:32462即可，如果看到如下欢迎界面证明 Kibana 已经成功部署到了Kubernetes集群之中。
#安装fluentd组件
我们使用daemonset控制器部署fluentd组件，这样可以保证集群中的每个节点都可以运行同样fluentd的pod副本，这样就可以收集k8s集群中每个节点的日志，在k8s集群中，容器应用程序的输入输出日志会重定向到node节点里的json文件中，fluentd可以tail和过滤以及把日志转换成指定的格式发送到elasticsearch集群中。除了容器日志，fluentd也可以采集kubelet、kube-proxy、docker的日志。
cat fluentd.yaml
apiVersion: v1 kind: ServiceAccount metadata: name: fluentd namespace: kube-logging labels: app: fluentd --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: fluentd labels: app: fluentd rules: - apiGroups: - "" resources: - pods - namespaces verbs: - get - list - watch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluentd roleRef: kind: ClusterRole name: fluentd apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: fluentd namespace: kube-logging --- apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd namespace: kube-logging labels: app: fluentd spec: selector: matchLabels: app: fluentd template: metadata: labels: app: fluentd spec: serviceAccount: fluentd serviceAccountName: fluentd tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:v1.4.2-debian-elasticsearch-1.1 imagePullPolicy: IfNotPresent env: - name: FLUENT_ELASTICSEARCH_HOST value: "elasticsearch.kube-logging.svc.cluster.local" - name: FLUENT_ELASTICSEARCH_PORT value: "9200" - name: FLUENT_ELASTICSEARCH_SCHEME value: "http" - name: FLUENTD_SYSTEMD_CONF value: disable resources: limits: memory: 512Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers
kubectl apply -f fluentd.yaml
查看是否部署成功
kubectl get pods -n kube-logging
显示如下，看到status状态是running，说明部署成功：
NAME READY STATUS RESTARTS AGE es-cluster-0 1/1 Running 6 57m es-cluster-1 1/1 Running 5 57m es-cluster-2 1/1 Running 0 45m fluentd-fs54n 1/1 Running 0 37m fluentd-ghgqf 1/1 Running 0 37m kibana-5749b5778b-zzgbc 1/1 Running 0 39m
Fluentd启动成功后，我们可以前往 Kibana 的 Dashboard 页面中，点击左侧的Discover，可以看到如下配置页面：
在这里可以配置我们需要的 Elasticsearch 索引，前面 Fluentd 配置文件中我们采集的日志使用的是 logstash 格式，这里只需要在文本框中输入logstash-*即可匹配到 Elasticsearch集群中的所有日志数据，然后点击下一步，进入以下页面：
点击next step，出现如下
选择@timestamp，创建索引
点击左侧的discover，可看到如下：
#测试容器日志
cat pod.yaml
apiVersion: v1 kind: Pod metadata: name: counter spec: containers: - name: count image: busybox imagePullPolicy: IfNotPresent args: [/bin/sh, -c,'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']
kubectl apply -f pod.yaml
登录到kibana的控制面板，在discover处的搜索栏中输入kubernetes.pod_name:counter，这将过滤名为的Pod的日志数据counter，如下所示：
总结
通过上面几个步骤，我们已经在k8s集群成功部署了elasticsearch，fluentd，kibana，这里使用的efk系统包括3个Elasticsearch Pod，一个Kibana Pod和一组作为DaemonSet部署的Fluentd Pod。
要了解更多关于elasticsearch可参考：https://www.elastic.co/cn/blog/small-medium-or-large-scaling-elasticsearch-and-evolving-the-elastic-stack-to-fit。
Kubernetes中还允许使用更复杂的日志系统，要了解更多信息，可参考https://kubernetes.io/docs/concepts/cluster-administration/logging/
