如果想要查看索引的慢日志,ES on PaaS中需要开启log4j配置,本文档主要说明如何手动修改ES配置加载log4j配置文件实现开启慢日志。
备注:官方文档参考 https://www.elastic.co/guide/en/elasticsearch/reference/6.8/logging.html
步骤:
1、ES开启慢日志配置
2、创建log4j2.properties的configmap配置文件
3、修改ES配置,添加log4j2的volumeMounts配置,触发ES重启
4、查看pod中,logs目录下的慢日志文件
详细操作:
1、ES开启慢日志配置
官方文档参考:
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/index-modules-slowlog.html
(1) 开启单个索引的慢日志配置,可根据需要调整慢日志的时间:
PUT /twitter/_settings
{
"index.search.slowlog.threshold.query.warn": "10s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.query.debug": "2s",
"index.search.slowlog.threshold.query.trace": "500ms",
"index.search.slowlog.threshold.fetch.warn": "1s",
"index.search.slowlog.threshold.fetch.info": "800ms",
"index.search.slowlog.threshold.fetch.debug": "500ms",
"index.search.slowlog.threshold.fetch.trace": "200ms",
"index.search.slowlog.level": "debug",
"index.indexing.slowlog.threshold.index.warn": "10s",
"index.indexing.slowlog.threshold.index.info": "5s",
"index.indexing.slowlog.threshold.index.debug": "2s",
"index.indexing.slowlog.threshold.index.trace": "500ms",
"index.indexing.slowlog.level": "debug",
"index.indexing.slowlog.source": "1000"
}
2、创建log4j2.properties的configmap配置文件
创建log4j2的configmap配置文件,其中关于日志的级别和清理规则可以自定义,namespace的名字要和ES实例的namespace保持一致,不然找不到configmap。
(注意,每个值的后面不要有空格,不然加载到ES中配置不生效)
执行 kubectl apply -f log4j2.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: log4j2
namespace: xxx-test
data:
log4j2.properties: |
appender.index_search_slowlog_rolling.type = RollingFile
appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling
appender.index_search_slowlog_rolling.fileName = ${
sys:es.logs.base_path}${
sys:file.separator}${
sys:es.logs.cluster_name}_index_search_slowlog.log
appender.index_search_slowlog_rolling.layout.type = PatternLayout
appender.index_search_slowlog_rolling.layout.pattern = [%d{
ISO8601}][%-5p][%-25c{
1.}] [%node_name]%marker %.-10000m%n
appender.index_search_slowlog_rolling.filePattern = ${
sys:es.logs.base_path}${
sys:file.separator}${
sys:es.logs.cluster_name}_index_search_slowlog-%d{
yyyy-MM-dd}-%i.log.gz
appender.index_search_slowlog_rolling.policies.type = Policies
appender.index_search_slowlog_rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.index_search_slowlog_rolling.policies.time.interval = 1
appender.index_search_slowlog_rolling.policies.time.modulate = true
appender.index_search_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.index_search_slowlog_rolling.policies.size.size = 256MB
appender.index_search_slowlog_rolling.strategy.type = DefaultRolloverStrategy
appender.index_search_slowlog_rolling.strategy.fileIndex = nomax
appender.index_search_slowlog_rolling.strategy.action.type = Delete
appender.index_search_slowlog_rolling.strategy.action.basepath = ${
sys:es.logs.base_path}${
sys:file.separator}
appender.index_search_slowlog_rolling.strategy.action.condition.type = IfFileName
appender.index_search_slowlog_rolling.strategy.action.condition.glob = ${
sys:es.logs.cluster_name}_*
appender.index_search_slowlog_rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
appender.index_search_slowlog_rolling.strategy.action.condition.nested_condition.exceeds = 2GB
logger.index_search_slowlog_rolling.name = index.search.slowlog
logger.index_search_slowlog_rolling.level = debug
logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling
logger.index_search_slowlog_rolling.additivity = false
appender.index_indexing_slowlog_rolling.type = RollingFile
appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling
appender.index_indexing_slowlog_rolling.fileName = ${
sys:es.logs.base_path}${
sys:file.separator}${
sys:es.logs.cluster_name}_index_indexing_slowlog.log
appender.index_indexing_slowlog_rolling.layout.type = PatternLayout
appender.index_indexing_slowlog_rolling.layout.pattern = [%d{
ISO8601}][%-5p][%-25c{
1.}] [%node_name]%marker %.-10000m%n
appender.index_indexing_slowlog_rolling.filePattern = ${
sys:es.logs.base_path}${
sys:file.separator}${
sys:es.logs.cluster_name}_index_indexing_slowlog-%d{
yyyy-MM-dd}-%i.log.gz
appender.index_indexing_slowlog_rolling.policies.type = Policies
appender.index_indexing_slowlog_rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.index_indexing_slowlog_rolling.policies.time.interval = 1
appender.index_indexing_slowlog_rolling.policies.time.modulate = true
appender.index_indexing_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.index_indexing_slowlog_rolling.policies.size.size = 256MB
appender.index_indexing_slowlog_rolling.strategy.type = DefaultRolloverStrategy
appender.index_indexing_slowlog_rolling.strategy.fileIndex = nomax
appender.index_indexing_slowlog_rolling.strategy.action.type = Delete
appender.index_indexing_slowlog_rolling.strategy.action.basepath = ${
sys:es.logs.base_path}${
sys:file.separator}
appender.index_indexing_slowlog_rolling.strategy.action.condition.type = IfFileName
appender.index_indexing_slowlog_rolling.strategy.action.condition.glob = ${
sys:es.logs.cluster_name}_*
appender.index_indexing_slowlog_rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
appender.index_indexing_slowlog_rolling.strategy.action.condition.nested_condition.exceeds = 2GB
logger.index_indexing_slowlog.name = index.indexing.slowlog.index
logger.index_indexing_slowlog.level = debug
logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling
logger.index_indexing_slowlog.additivity = false
3、修改ES配置,添加log4j2的volumeMounts配置
kubectl get es -A
kubectl edit es esxxx -n xxx
在data和master的dataVolumeClaim同级目录下添加如下内容:
(备注:如果是粘贴复制方式,先执行:set paste , 再黏贴,这样可以避免空格的影响
podTemplate:
spec:
containers:
- name: es
volumeMounts:
- name: log4j2
mountPath: /usr/share/elasticsearch/config/log4j2.properties
subPath: log4j2.properties
volumes:
- name: log4j2
configMap:
name: log4j2
保存更改后,注意查看ES的rolling状态
kubect get es -A
4、查看pod中,logs目录下的慢日志文件
kubectl exec -it es01-data-0 -n xxx-test bash
注意:Elasticsearch默认关闭慢日志,开启可能会对性能造成影响。