es-sts-deploy.yaml 集群部署清单：
cat << EOF > es-sts-deploy.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: es-cluster namespace: kube-logging spec: serviceName: elasticsearch replicas: 3 selector: matchLabels: app: elasticsearch template: metadata: labels: app: elasticsearch spec: containers: - name: elasticsearch image: elasticsearch:7.8.0 imagePullPolicy: IfNotPresent resources: limits: cpu: 1000m requests: cpu: 100m ports: - containerPort: 9200 name: rest protocol: TCP - containerPort: 9300 name: inter-node protocol: TCP volumeMounts: - name: data mountPath: /usr/share/elasticsearch/data env: - name: cluster.name value: k8s-logs - name: node.name valueFrom: fieldRef: fieldPath: metadata.name - name: discovery.seed_hosts value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch" - name: cluster.initial_master_nodes value: "es-cluster-0,es-cluster-1,es-cluster-2" - name: ES_JAVA_OPTS value: "-Xms512m -Xmx512m" initContainers: - name: fix-permissions image: busybox imagePullPolicy: IfNotPresent command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"] securityContext: privileged: true volumeMounts: - name: data mountPath: /usr/share/elasticsearch/data - name: increase-vm-max-map image: busybox imagePullPolicy: IfNotPresent command: ["sysctl", "-w", "vm.max_map_count=262144"] securityContext: privileged: true - name: increase-fd-ulimit image: busybox imagePullPolicy: IfNotPresent command: ["sh", "-c", "ulimit -n 65536"] securityContext: privileged: true volumeClaimTemplates: - metadata: name: data labels: app: elasticsearch spec: accessModes: [ "ReadWriteOnce" ] storageClassName: managed-nfs-storage resources: requests: storage: 10Gi EOF
OK，稍等几分钟后，es集群基本就部署好了，看看pod和svc是否正常吧：
[root@k8s-master ~]# k get po,svc -n kube-logging NAME READY STATUS RESTARTS AGE pod/es-cluster-0 1/1 Running 0 6m44s pod/es-cluster-1 1/1 Running 0 6m37s pod/es-cluster-2 1/1 Running 0 6m30s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 3m51s
四，kibana的部署
这个没什么好说的，干就完了，需要注意的是，镜像和上面的es版本一致，都是7.8.0哦。
总共两个清单文件，一个是service，该service是暴露节点端口的，如果有安装ingress，那么，此service可以设置为headless service不用设置为NodePort。
第二个文件是部署pod的文件，其中的value: http://elasticsearch:9200 是指的headless service的9200端口，假如啊，注意我这是假如，headless service名字是myes，那么，这里value就应该是 http://myes:9200，总之，此环境变量把kibana和elasticsearch集群联系起来了。
kibana-svc.yaml
cat << EOF > kibana-svc.yaml apiVersion: v1 kind: Service metadata: name: kibana namespace: kube-logging labels: app: kibana spec: type: NodePort ports: - port: 5601 selector: app: kibana EOF
kibana-deploy.yaml
cat << EOF > kibana-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: kibana namespace: kube-logging labels: app: kibana spec: replicas: 1 selector: matchLabels: app: kibana template: metadata: labels: app: kibana spec: containers: - name: kibana image: docker.elastic.co/kibana/kibana:7.8.0 imagePullPolicy: IfNotPresent resources: limits: cpu: 1000m requests: cpu: 100m env: - name: ELASTICSEARCH_URL value: http://elasticsearch:9200 ports: - containerPort: 5601 EOF
稍等大概5分钟，查看一哈kibana的日志 ，直到有这个出现：http server running at http://0:5601 表示kibana部署完成。
","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2022-10-15T11:45:36Z","tags":["info","plugins","taskManager","taskManager"],"pid":6,"message":"TaskManager is identified by the Kibana UUID: dd9bcb6f-4353-4861-81e5-fe3ac42bb157"} {"type":"log","@timestamp":"2022-10-15T11:45:36Z","tags":["status","plugin:task_manager@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2022-10-15T11:45:36Z","tags":["status","plugin:encryptedSavedObjects@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2022-10-15T11:45:36Z","tags":["status","plugin:apm_oss@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2022-10-15T11:45:36Z","tags":["status","plugin:console_legacy@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2022-10-15T11:45:36Z","tags":["status","plugin:region_map@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2022-10-15T11:45:36Z","tags":["status","plugin:ui_metric@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2022-10-15T11:45:36Z","tags":["listening","info"],"pid":6,"message":"Server running at http://0:5601"} {"type":"log","@timestamp":"2022-10-15T11:45:38Z","tags":["info","http","server","Kibana"],"pid":6,"message":"http server running at http://0:5601"}
看一下kibana相关的pod和service是否正常：
[root@k8s-master ~]# k get po,svc -n kube-logging NAME READY STATUS RESTARTS AGE pod/es-cluster-0 1/1 Running 0 12m pod/es-cluster-1 1/1 Running 0 12m pod/es-cluster-2 1/1 Running 0 12m pod/kibana-588d597485-wljbr 1/1 Running 0 49s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 9m21s service/kibana NodePort 10.0.132.94 <none> 5601:32042/TCP 49s
打开浏览器，任意一个节点IP+32042就可以登录kibana了。
五，采集器fluentd的部署
ＳｅｒｖｉｃｅＡｃｃｅｎｔ清单文件：
cat << EOF > fluentd-sa.yaml apiVersion: v1 kind: ServiceAccount metadata: name: fluentd namespace: kube-logging labels: app: fluentd EOF
ｆｌｕｅｎｔｄ的ｒｂａｃ：
cat << EOF > fluentd-rbac.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: fluentd labels: app: fluentd rules: - apiGroups: - "" resources: - pods - namespaces verbs: - get - list - watch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluentd roleRef: kind: ClusterRole name: fluentd apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: fluentd namespace: kube-logging EOF
ｆｌｕｅｎｔｄ的ＤａｅｍｏｎＳｅｔ部署清单 ：
配置说明：
将宿主机Node的/var/log和/var/lib/docker/containers目录挂载到 fluentd容器中，用于读取容器输出到stdout和stderr的日志，以及kubernetes组件的日志。
资源限制根据实际情况进行调整，避免Fluentd占用太多资源。
利用环境变量，设置了elasticsarch服务的访问地址，此处使用了service名称，也就是这个：elasticsearch.kube-logging.svc.cluster.local。
cat << EOF > fluentd-deploy.yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd namespace: kube-logging labels: app: fluentd spec: selector: matchLabels: app: fluentd template: metadata: labels: app: fluentd spec: serviceAccount: fluentd serviceAccountName: fluentd tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:v1.4.2-debian-elasticsearch-1.1 imagePullPolicy: IfNotPresent env: - name: FLUENT_ELASTICSEARCH_HOST value: "elasticsearch.kube-logging.svc.cluster.local" - name: FLUENT_ELASTICSEARCH_PORT value: "9200" - name: FLUENT_ELASTICSEARCH_SCHEME value: "http" - name: FLUENTD_SYSTEMD_CONF value: disable resources: limits: memory: 512Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers EOF
那么，fluent的变量有哪些呢？进入pod里，查看fluent的配置文件，里面有比如 FLUENT_ELASTICSEARCH_HOST，ENV['FLUENT_ELASTICSEARCH_PORT以及日志等级log_level 这里是info等等变量。
root@fluentd-d58br:/fluentd/etc# cat fluent.conf # AUTOMATICALLY GENERATED # DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/fluent.conf.erb @include "#{ENV['FLUENTD_SYSTEMD_CONF'] || 'systemd'}.conf" @include "#{ENV['FLUENTD_PROMETHEUS_CONF'] || 'prometheus'}.conf" @include kubernetes.conf @include conf.d/*.conf <match **> @type elasticsearch @id out_es @log_level info include_tag_key true host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}" port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}" path "#{ENV['FLUENT_ELASTICSEARCH_PATH']}" scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}" ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}" ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1'}" reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'false'}" reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'true'}" reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'true'}" log_es_400_reason "#{ENV['FLUENT_ELASTICSEARCH_LOG_ES_400_REASON'] || 'false'}" logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}" logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'true'}" index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'logstash'}" type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'fluentd'}" <buffer> flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}" flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}" chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}" queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}" retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}" retry_forever true </buffer> </match>
