搜索+聚合：统计指定品牌下每个颜色的销量
实际上来说，我们之前学习的搜索相关的知识，完全可以和聚合组合起来使用
select count(*)
from tvs.sales
where brand like "%长%"
group by price
es aggregation，scope，任何的聚合，都必须在搜索出来的结果数据中之行，搜索结果，就是聚合分析操作的scope
GET /tvs/sales/_search { "size": 0, "query": { "term": { "brand": { "value": "小米" } } }, "aggs": { "group_by_color": { "terms": { "field": "color" } } } }
{ "took": 5, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 2, "max_score": 0, "hits": [] }, "aggregations": { "group_by_color": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "绿色", "doc_count": 1 }, { "key": "蓝色", "doc_count": 1 } ] } } }
cardinality去重算法以及每月销售品牌数量统计
cartinality metric，对每个bucket中的指定的field进行去重，取去重后的count，类似于count(distcint)
GET /tvs/sales/_search { "size" : 0, "aggs" : { "months" : { "date_histogram": { "field": "sold_date", "interval": "month" }, "aggs": { "distinct_colors" : { "cardinality" : { "field" : "brand" } } } } } }
{ "took": 70, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 8, "max_score": 0, "hits": [] }, "aggregations": { "group_by_sold_date": { "buckets": [ { "key_as_string": "2016-05-01T00:00:00.000Z", "key": 1462060800000, "doc_count": 1, "distinct_brand_cnt": { "value": 1 } }, { "key_as_string": "2016-06-01T00:00:00.000Z", "key": 1464739200000, "doc_count": 0, "distinct_brand_cnt": { "value": 0 } }, { "key_as_string": "2016-07-01T00:00:00.000Z", "key": 1467331200000, "doc_count": 1, "distinct_brand_cnt": { "value": 1 } }, { "key_as_string": "2016-08-01T00:00:00.000Z", "key": 1470009600000, "doc_count": 1, "distinct_brand_cnt": { "value": 1 } }, { "key_as_string": "2016-09-01T00:00:00.000Z", "key": 1472688000000, "doc_count": 0, "distinct_brand_cnt": { "value": 0 } }, { "key_as_string": "2016-10-01T00:00:00.000Z", "key": 1475280000000, "doc_count": 1, "distinct_brand_cnt": { "value": 1 } }, { "key_as_string": "2016-11-01T00:00:00.000Z", "key": 1477958400000, "doc_count": 2, "distinct_brand_cnt": { "value": 1 } }, { "key_as_string": "2016-12-01T00:00:00.000Z", "key": 1480550400000, "doc_count": 0, "distinct_brand_cnt": { "value": 0 } }, { "key_as_string": "2017-01-01T00:00:00.000Z", "key": 1483228800000, "doc_count": 1, "distinct_brand_cnt": { "value": 1 } }, { "key_as_string": "2017-02-01T00:00:00.000Z", "key": 1485907200000, "doc_count": 1, "distinct_brand_cnt": { "value": 1 } } ] } } }
percentiles百分比算法以及网站访问时延统计
需求：比如有一个网站，记录下了每次请求的访问的耗时，需要统计tp50，tp90，tp99
tp50：50%的请求的耗时最长在多长时间
tp95：95%的请求的耗时最长在多长时间
tp99：99%的请求的耗时最长在多长时间
PUT /website { "mappings": { "logs": { "properties": { "latency": { "type": "long" }, "province": { "type": "keyword" }, "timestamp": { "type": "date" } } } } }
POST /website/logs/_bulk { "index": {}} { "latency" : 105, "province" : "江苏", "timestamp" : "2016-10-28" } { "index": {}} { "latency" : 83, "province" : "江苏", "timestamp" : "2016-10-29" } { "index": {}} { "latency" : 92, "province" : "江苏", "timestamp" : "2016-10-29" } { "index": {}} { "latency" : 112, "province" : "江苏", "timestamp" : "2016-10-28" } { "index": {}} { "latency" : 68, "province" : "江苏", "timestamp" : "2016-10-28" } { "index": {}} { "latency" : 76, "province" : "江苏", "timestamp" : "2016-10-29" } { "index": {}} { "latency" : 101, "province" : "新疆", "timestamp" : "2016-10-28" } { "index": {}} { "latency" : 275, "province" : "新疆", "timestamp" : "2016-10-29" } { "index": {}} { "latency" : 166, "province" : "新疆", "timestamp" : "2016-10-29" } { "index": {}} { "latency" : 654, "province" : "新疆", "timestamp" : "2016-10-28" } { "index": {}} { "latency" : 389, "province" : "新疆", "timestamp" : "2016-10-28" } { "index": {}} { "latency" : 302, "province" : "新疆", "timestamp" : "2016-10-29" }
pencentiles
GET /website/logs/_search { "size": 0, "aggs": { "latency_percentiles": { "percentiles": { "field": "latency", "percents": [ 50, 95, 99 ] } }, "latency_avg": { "avg": { "field": "latency" } } } }
{ "took": 31, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 12, "max_score": 0, "hits": [] }, "aggregations": { "latency_avg": { "value": 201.91666666666666 }, "latency_percentiles": { "values": { "50.0": 108.5, "95.0": 508.24999999999983, "99.0": 624.8500000000001 } } } }
返回字段的含义：50%的请求，在108.5ms内，不是完全准确的使用一定的算法算出来的
GET /website/logs/_search { "size": 0, "aggs": { "group_by_province": { "terms": { "field": "province" }, "aggs": { "latency_percentiles": { "percentiles": { "field": "latency", "percents": [ 50, 95, 99 ] } }, "latency_avg": { "avg": { "field": "latency" } } } } } }
{ "took": 33, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 12, "max_score": 0, "hits": [] }, "aggregations": { "group_by_province": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "新疆", "doc_count": 6, "latency_avg": { "value": 314.5 }, "latency_percentiles": { "values": { "50.0": 288.5, "95.0": 587.75, "99.0": 640.75 } } }, { "key": "江苏", "doc_count": 6, "latency_avg": { "value": 89.33333333333333 }, "latency_percentiles": { "values": { "50.0": 87.5, "95.0": 110.25, "99.0": 111.65 } } } ] } } }
string field聚合实验以及fielddata原理初探
1、对于分词的field执行aggregation，发现报错。。。
GET /test_index/test_type/_search { "aggs": { "group_by_test_field": { "terms": { "field": "test_field" } } } }
{ "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [test_field] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory." } ], "type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query", "grouped": true, "failed_shards": [ { "shard": 0, "index": "test_index", "node": "4onsTYVZTjGvIj9_spWz2w", "reason": { "type": "illegal_argument_exception", "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [test_field] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory." } } ], "caused_by": { "type": "illegal_argument_exception", "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [test_field] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory." } }, "status": 400 }
对分词的field，直接执行聚合操作，会报错，大概意思是说，你必须要打开fielddata，然后将正排索引数据加载到内存中，才可以对分词的field执行聚合操作，而且会消耗很大的内存
2、给分词的field，设置fielddata=true，发现可以执行，但是结果却。。。
POST /test_index/_mapping/test_type { "properties": { "test_field": { "type": "text", "fielddata": true } } } { "test_index": { "mappings": { "test_type": { "properties": { "test_field": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "fielddata": true } } } } } }
GET /test_index/test_type/_search { "size": 0, "aggs": { "group_by_test_field": { "terms": { "field": "test_field" } } } }
{ "took": 23, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 2, "max_score": 0, "hits": [] }, "aggregations": { "group_by_test_field": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "test", "doc_count": 2 } ] } } }
如果要对分词的field执行聚合操作，必须将fielddata设置为true
3、使用内置field不分词，对string field进行聚合
GET /test_index/test_type/_search { "size": 0, "aggs": { "group_by_test_field": { "terms": { "field": "test_field.keyword" } } } }
{ "took": 3, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 2, "max_score": 0, "hits": [] }, "aggregations": { "group_by_test_field": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "test", "doc_count": 2 } ] } } }
如果对不分词的field执行聚合操作，直接就可以执行，不需要设置fieldata=true
4、分词field+fielddata的工作原理
doc value --> 不分词的所有field，可以执行聚合操作 --> 如果你的某个field不分词，那么在index-time，就会自动生成doc value --> 针对这些不分词的field执行聚合操作的时候，自动就会用doc value来执行
分词field，是没有doc value的。。。在index-time，如果某个field是分词的，那么是不会给它建立doc value正排索引的，因为分词后，占用的空间过于大，所以默认是不支持分词field进行聚合的
分词field默认没有doc value，所以直接对分词field执行聚合操作，是会报错的
对于分词field，必须打开和使用fielddata，完全存在于纯内存中。。。结构和doc value类似。。。如果是ngram或者是大量term，那么必将占用大量的内存。。。
如果一定要对分词的field执行聚合，那么必须将fielddata=true，然后es就会在执行聚合操作的时候，现场将field对应的数据，建立一份fielddata正排索引，fielddata正排索引的结构跟doc value是类似的，但是只会将fielddata正排索引加载到内存中来，然后基于内存中的fielddata正排索引执行分词field的聚合操作
如果直接对分词field执行聚合，报错，才会让我们开启fielddata=true，告诉我们，会将fielddata uninverted index，正排索引，加载到内存，会耗费内存空间
为什么fielddata必须在内存？因为大家自己思考一下，分词的字符串，需要按照term进行聚合，需要执行更加复杂的算法和操作，如果基于磁盘和os cache，那么性能会很差
