《Elastic Stack 实战手册》——三、产品能力——3.5 进阶篇——3.5.3.Rollup (3) https://developer.aliyun.com/article/1228768
原始数据和汇总索引同时查询实现原理：
Elasticsearch 接收到原始数据和汇总数据联合 _rollup_search 查询响应后， 会重写汇总响应，并将两者合并在一起。在合并过程中，如果两个响应之间的存储桶中有任何重叠，则使用非汇总索引中汇总的桶数据。
样例：
创建新的复杂任务，具体任务信息如下：
# 按照基础 API 步骤（通过 Kibana 更加简单），创建复杂任务，汇总多个指标，任务详情如下 { "config": { "id": "es-slowlog-agg-id1", "index_pattern": "es-slowlog*", "rollup_index": "rollup-es-slowlog-agg1", "cron": "0 * * * * ?", "groups": { "date_histogram": { "calendar_interval": "1m", "field": "timestamp_local", "delay": "1m", "time_zone": "UTC" }, "histogram": { "interval": 8, "fields": [ "event.duration" ] }, "terms": { "fields": [ "cluster", "elasticsearch.index.name", "host.name" ] } }, "metrics": [ { 982 > 三、产品能力 "field": "event.duration", "metrics": [ "avg", "max", "min", "sum", "value_count" ] } ], "timeout": "20s", "page_size": 10000 }, "status": { "job_state": "started", "current_position": { "cluster.terms": "clustername-demo", "elasticsearch.index.name.terms": "basiclog-slowlog_2021-04-02", "event.duration.histogram": 2307000000, "host.name.terms": "host_name-demo", "timestamp_local.date_histogram": 1618984980000 }, "upgraded_doc_id": true }, "stats": { "pages_processed": 6, "documents_processed": 1, "rollups_indexed": 1, "trigger_count": 5, "index_time_in_ms": 115, "index_total": 1, "index_failures": 0, "search_time_in_ms": 21, "search_total": 6, "search_failures": 0, "processing_time_in_ms": 0, "processing_total": 6 } }
_search 查询汇总目标索引中的原始数据：
GET rollup-es-slowlog-agg1/_search { "size":10, "query": { "bool": { "must": [], "filter": [ { "match_all": {} } ], "should": [], "must_not": [] } } } #返回结果 { "took": 2, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 1, "relation": "eq" }, "max_score": 1, "hits": [ { "_index": "rollup-es-slowlog-agg1", "_type": "_doc", "_id": "es-slowlog-agg-id1$5uzfGmyS2uAb3XRznkZBgA", "_score": 1, "_source": { "cluster.terms.value": "bj-ali-xueyan-oa-es-cluster", "event.duration.avg._count": 1, "event.duration.max.value": 2377000000, "event.duration.histogram.value": 2377000000, "timestamp_local.date_histogram.time_zone": "UTC", "elasticsearch.index.name.terms.value": "basiclog-slowlog_2400-2021-04-02", "host.name.terms._count": 1, "cluster.terms._count": 1, "host.name.terms.value": "bj-sjhl-university-es-online-99-62", "event.duration.avg.value": 2377000000, "elasticsearch.index.name.terms._count": 1, "event.duration.histogram.interval": 8, "timestamp_local.date_histogram._count": 1, "timestamp_local.date_histogram.timestamp": 1618995780000, "_rollup.version": 2, "event.duration.histogram._count": 1, "timestamp_local.date_histogram.interval": "1m", "event.duration.sum.value": 2377000000, "event.duration.min.value": 2377000000, "event.duration.value_count.value": 1, "_rollup.id": "es-slowlog-agg-id1" } } ] } }
_rollup_search 查询数据（可以把原始数据和汇总数据联合查询）
GET es-slowlog*,rollup-es-slowlog-agg1/_rollup_search { "size": 0, "aggregations": { "avg_event.duration": { "avg": { "field": "event.duration" } } } } #返回值 { "took": 740, "timed_out": false, "terminated_early": false, "num_reduce_phases": 2, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 0, "relation": "eq" }, "max_score": 0, "hits": [ ] }, "aggregations": { "avg_event.duration": { "value": 2311777445.714286 } } }
获取汇总信息
根据 Rollup 配置中的 index_pattern 获取对应的任务,支持 _all 查询所有。
请求：GET _rollup/data/
#查询所有 GET _rollup/data/_all #查询指定目标 GET _rollup/data/es-slowlog* { "es-slowlog*": { "rollup_jobs": [ { "job_id": "es-slowlog-agg-id", "rollup_index": "rollup-es-slowlog-agg", "index_pattern": "es-slowlog*", "fields": { "cluster": [ { "agg": "terms" } ], "timestamp_local": [ { "agg": "date_histogram", "delay": "1m", "time_zone": "UTC", "calendar_interval": "1m" } ], "elasticsearch.index.name": [ { "agg": "terms" } ], "host.name": [ { "agg": "terms" } ] } }, { "job_id": "es-slowlog-agg-id1", "rollup_index": "rollup-es-slowlog-agg", "index_pattern": "es-slowlog*", "fields": { "cluster": [ { "agg": "terms" } ], "timestamp_local": [ { "agg": "date_histogram", "delay": "1m", "time_zone": "UTC", "calendar_interval": "1m" } ], "elasticsearch.index.name": [ { "agg": "terms" } ], "host.name": [ { "agg": "terms" } ] } }, { "job_id": "es-slowlog-agg-id1", "rollup_index": "rollup-es-slowlog-agg1", "index_pattern": "es-slowlog*", "fields": { "event.duration": [ { "agg": "histogram", "interval": 8 }, { "agg": "avg" }, { "agg": "max" }, { "agg": "min" }, { "agg": "sum" }, { "agg": "value_count" } ], "cluster": [ { "agg": "terms" } ], "timestamp_local": [ { "agg": "date_histogram", "delay": "1m", "time_zone": "UTC", "calendar_interval": "1m" } ], "elasticsearch.index.name": [ { "agg": "terms" } ], "host.name": [ { "agg": "terms" } ] } }, { "job_id": "es-slowlog-agg-id3", "rollup_index": "rollupes-slowlog-agg", "index_pattern": "es-slowlog*", "fields": { "cluster": [ { "agg": "terms" } ], "timestamp_local": [ { "agg": "date_histogram", "delay": "1m", "time_zone": "UTC", "calendar_interval": "1m" } ], "elasticsearch.index.name": [ { "agg": "terms" } ], "host.name": [ { "agg": "terms" } ] } } ] } }
根据 Rollup 目标索引查询对应的任务，支持 * 匹配。
请求：GET /_rollup/data
GET rollupes-slowlog-*/_rollup/data GET rollupes-slowlog-agg/_rollup/data { "rollupes-slowlog-agg": { "rollup_jobs": [ { "job_id": "es-slowlog-agg-id3", "rollup_index": "rollupes-slowlog-agg", "index_pattern": "es-slowlog*", "fields": { "cluster": [ { "agg": "terms" } ], "timestamp_local": [ { "agg": "date_histogram", "delay": "1m", "time_zone": "UTC", "calendar_interval": "1m" } ], "elasticsearch.index.name": [ { "agg": "terms" } ], "host.name": [ { "agg": "terms" } ] } } ] } }
《Elastic Stack 实战手册》——三、产品能力——3.5 进阶篇——3.5.3.Rollup (5) https://developer.aliyun.com/article/1228766
