记一次测试环境kibana无日志
背景：
kubernetes集群，kibana集群没有数据
访问链路：filebeat-kafka-logstash-elasticsearch-kibana
场景：测试环境kafka只设置了单点。即broker只有一个
排查思路：
#登录kibana，查看ealsticsearch节点是否red或者在master执行 curl 127.0.0.1:port/_cat/health #查看pod状态是否正常 kubectl get pods -A #根据访问链路一个个查询log kubectl log -n namespace filebeat/kafka/logstash/elasticsearch/kibana #发现kafka有报错信息 error：1 partitions have leader brokers without a matching listener, including [baidd-0] (org.apache.kafka. 说是有一个分区出现了问题，
处理：
#进入kafka pod kubectl exec -it -n namespace kafka*** bash #cd bin kafka-topic.sh --zookeeper zookeeper:2181 --describe outlist： Topic:test-trace-log PartitionCount:1 ReplicationFactor:1 Configs: Topic: test-trace-log Partition: 0 Leader: -1 Replicas: 1001 Isr: 1001 发现leader：-1得知出现了单点故障，没有找到正确的borker #查询现在的broker id find / -name "meta.properties" 文件内输出： version=0 broker.id=1002 #进入zookeeper pod查看leader的id kubectl exec -it -n namespace zookeeper*** bash #cd bin ./zkCli.sh [zk: localhost:2181(CONNECTED) 2] ls /brokers/ids [1002] [zk: localhost:2181(CONNECTED) 3] get /brokers/topics/test-trace-log/partitions/0/state {"controller_epoch":20,"leader":-1,"version":1,"leader_epoch":5,"isr":[1001]} #使用set修改leader和isr，修改的值要是存在的值，例如brokerid=1002 set /brokers/topics/test-trace-log/partitions/0/state {"controller_epoch":20,"leader":1002,"version":1,"leader_epoch":5,"isr":[1002]} #必须重启kafka pod**** 在kafka pod内操作 #Replicas: 1001 1001这个在单节点上是不存在的，一样会没有数据，需要改为单点brokerid =1002 vim replicas.json { "version": 1, "partitions": [{ "topic": "test-trace-log", "partition": 0, "replicas": [1002] }] } 一个topic多个分区可以批量操作： { "version": 1, "partitions": [ { "topic": "test-trace-log", "partition": 0, "replicas": [1002] }, { "topic": "test-trace-log", "partition": 1, "replicas": [1002] } ] } #cd bin kafka-reassign-partitions.sh --zookeeper zookeeper:2181 --reassignment-json-file replicas.json --execute #查看此时的topic info #cd bin kafka-topics.sh --zookeeper zookeeper:2181 --describe Topic:test-trace-log PartitionCount:1 ReplicationFactor:1 Configs: Topic: test-trace-log Partition: 0 Leader: 1002 Replicas: 1002 Isr: 1002
查看kibana，已经有日志输出。
