2.10 布尔查询-JavaAPI-对多个查询条件连接
布尔查询：boolQuery
- 查询品牌名称为:华为
- 查询标题包含：手机
- 查询价格在：2000-3000
must 、filter为连接方式
term、match为不同的查询方式
//1.构建boolQuery BoolQueryBuilder boolQuery = QueryBuilders.boolQuery(); //2.构建各个查询条件 //2.1 查询品牌名称为:华为 TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("brandName", "华为"); boolQuery.must(termQueryBuilder); //2.2. 查询标题包含：手机 MatchQueryBuilder matchQuery = QueryBuilders.matchQuery("title", "手机"); boolQuery.filter(matchQuery); //2.3 查询价格在：2000-3000 RangeQueryBuilder rangeQuery = QueryBuilders.rangeQuery("price"); rangeQuery.gte(2000); rangeQuery.lte(3000); boolQuery.filter(rangeQuery); sourceBuilder.query(boolQuery);
2.11 聚合查询-脚本
•指标聚合：相当于MySQL的聚合函数。max、min、avg、sum等
•桶聚合：相当于MySQL的 group by 操作。不要对text类型的数据进行分组，会失败。
# 聚合查询 # 指标聚合 聚合函数 GET goods/_search { "query": { "match": { "title": "手机" } }, "aggs": { "max_price": { "max": { "field": "price" } } } } # 桶聚合 分组 GET goods/_search { "query": { "match": { "title": "手机" } }, "aggs": { "goods_brands": { "terms": { "field": "brandName", "size": 100 } } } }
2.12 聚合查询-JavaAPI
聚合查询：桶聚合，分组查询
- 查询title包含手机的数据
- 查询品牌列表
/** * 聚合查询：桶聚合，分组查询 * 1. 查询title包含手机的数据 * 2. 查询品牌列表 */ @Test public void testAggQuery() throws IOException { SearchRequest searchRequest=new SearchRequest("goods"); SearchSourceBuilder sourceBuilder=new SearchSourceBuilder(); //1. 查询title包含手机的数据 MatchQueryBuilder queryBuilder = QueryBuilders.matchQuery("title", "手机"); sourceBuilder.query(queryBuilder); //2. 查询品牌列表 只展示前100条 AggregationBuilder aggregation=AggregationBuilders.terms("goods_brands").field("brandName").size(100); sourceBuilder.aggregation(aggregation); searchRequest.source(sourceBuilder); SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT); //7. 获取命中对象 SearchHits SearchHits hits = searchResponse.getHits(); //7.1 获取总记录数 Long total= hits.getTotalHits().value; System.out.println("总数："+total); // aggregations 对象 Aggregations aggregations = searchResponse.getAggregations(); //将aggregations 转化为map Map<String, Aggregation> aggregationMap = aggregations.asMap(); //通过key获取goods_brands 对象 使用Aggregation的子类接收 buckets属性在Terms接口中体现 // Aggregation goods_brands1 = aggregationMap.get("goods_brands"); Terms goods_brands =(Terms) aggregationMap.get("goods_brands"); //获取buckets 数组集合 List<? extends Terms.Bucket> buckets = goods_brands.getBuckets(); Map<String,Object>map=new HashMap<>(); //遍历buckets key 属性名，doc_count 统计聚合数 for (Terms.Bucket bucket : buckets) { System.out.println(bucket.getKey()); map.put(bucket.getKeyAsString(),bucket.getDocCount()); } System.out.println(map); }
2.13 高亮查询-脚本
高亮三要素：
•高亮字段
•前缀
•后缀
默认前后缀 ：em
<em>手机</em>
GET goods/_search { "query": { "match": { "title": "电视" } }, "highlight": { "fields": { "title": { "pre_tags": "<font color='red'>", "post_tags": "</font>" } } } }
2.14 高亮查询-JavaAPI
实施步骤：
高亮查询：
1. 设置高亮
高亮字段
前缀
后缀
2. 将高亮了的字段数据，替换原有数据
/** * * 高亮查询： * 1. 设置高亮 * * 高亮字段 * * 前缀 * * 后缀 * 2. 将高亮了的字段数据，替换原有数据 */ @Test public void testHighLightQuery() throws IOException { SearchRequest searchRequest = new SearchRequest("goods"); SearchSourceBuilder sourceBulider = new SearchSourceBuilder(); // 1. 查询title包含手机的数据 MatchQueryBuilder query = QueryBuilders.matchQuery("title", "手机"); sourceBulider.query(query); //设置高亮 HighlightBuilder highlighter = new HighlightBuilder(); //设置三要素 highlighter.field("title"); //设置前后缀标签 highlighter.preTags("<font color='red'>"); highlighter.postTags("</font>"); //加载已经设置好的高亮配置 sourceBulider.highlighter(highlighter); searchRequest.source(sourceBulider); SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT); SearchHits searchHits = searchResponse.getHits(); //获取记录数 long value = searchHits.getTotalHits().value; System.out.println("总记录数："+value); List<Goods> goodsList = new ArrayList<>(); SearchHit[] hits = searchHits.getHits(); for (SearchHit hit : hits) { String sourceAsString = hit.getSourceAsString(); //转为java Goods goods = JSON.parseObject(sourceAsString, Goods.class); // 获取高亮结果，替换goods中的title Map<String, HighlightField> highlightFields = hit.getHighlightFields(); HighlightField HighlightField = highlightFields.get("title"); Text[] fragments = HighlightField.fragments(); //highlight title替换 替换goods中的title goods.setTitle(fragments[0].toString()); goodsList.add(goods); } for (Goods goods : goodsList) { System.out.println(goods); } }
2.15 重建索引&索引别名
#查询别名 默认别名无法查看，默认别名同索引名 GET goods/_alias/ #结果 { "goods" : { "aliases" : { } } }
1.新建student_index_v1索引
# -------重建索引----------- # 新建student_index_v1。索引名称必须全部小写 PUT student_index_v1 { "mappings": { "properties": { "birthday":{ "type": "date" } } } } #查看 student_index_v1 结构 GET student_index_v1 #添加数据 PUT student_index_v1/_doc/1 { "birthday":"1999-11-11" } #查看数据 GET student_index_v1/_search #添加数据 PUT student_index_v1/_doc/1 { "birthday":"1999年11月11日" }
2.重建索引:将student_index_v1 数据拷贝到 student_index_v2
# 业务变更了，需要改变birthday字段的类型为text # 1. 创建新的索引 student_index_v2 # 2. 将student_index_v1 数据拷贝到 student_index_v2 # 创建新的索引 student_index_v2 PUT student_index_v2 { "mappings": { "properties": { "birthday":{ "type": "text" } } } } # 将student_index_v1 数据拷贝到 student_index_v2 # _reindex 拷贝数据 POST _reindex { "source": { "index": "student_index_v1" }, "dest": { "index": "student_index_v2" } } GET student_index_v2/_search PUT student_index_v2/_doc/2 { "birthday":"1999年11月11日" }
3.创建索引库别名：
注意：DELETE student_index_v1 这一操作将删除student_index_v1索引库，并不是删除别名
# 思考： 现在java代码中操作es，还是使用的实student_index_v1老的索引名称。 # 1. 改代码（不推荐） # 2. 索引别名（推荐） # 步骤： # 0. 先删除student_index_v1 # 1. 给student_index_v2起个别名 student_index_v1 # 先删除student_index_v1 #DELETE student_index_v1 这一操作将删除student_index_v1索引库 #索引库默认的别名与索引库同名，无法删除 # 给student_index_v1起个别名 student_index_v11 POST student_index_v2/_alias/student_index_v11 #测试删除命令 POST /_aliases { "actions": [ {"remove": {"index": "student_index_v1", "alias": "student_index_v11"}} ] } # 给student_index_v2起个别名 student_index_v1 POST student_index_v2/_alias/student_index_v1 #查询别名 GET goods/_alias/ GET student_index_v1/_search GET student_index_v2/_search
3 ES复杂聚合查询
3.1 统计字段总数
@Override public PersonStaticDTO getUserAgg(UserQuery userQuery) { SearchQuery query = new NativeSearchQueryBuilder() .withQuery(userMapper.getUserQueryBuilder(userQuery)) //统计这个字段的总数 .addAggregation(AggregationBuilders.count("userCount").field("accountId")) .build(); AggregatedPage<User> page = elasticsearchTemplate.queryForPage(query, User.class); //之后获得这个统计对象 进行赋值 ValueCount userCount = (ValueCount) page.getAggregation("userCount"); PersonStaticDTO personStaticDTO = new PersonStaticDTO(); personStaticDTO.setCount(((Double)userCount.value()).intValue()); return personStaticDTO; }
3.2 枚举分组统计总数
@Override public DeviceRepairCountDTO getRepairAgg(OperatorDeviceRepairQueryDTO operatorDeviceRepairQueryDTO) { SearchQuery query = new NativeSearchQueryBuilder() .withQuery(RepairQueryUtils.getRepairQueryBuilder(operatorDeviceRepairQueryDTO)) //通过状态进行分组 .addAggregation(AggregationBuilders.filter("Pending", QueryBuilders.termQuery("repairStatus", RepairStatus.Pending.ordinal()))) .addAggregation(AggregationBuilders.filter("Processing", QueryBuilders.termQuery("repairStatus", RepairStatus.Processing.ordinal()))) .withPageable(PageRequest.of(0, 1)) .build(); AggregatedPage<DeviceRepairRecordDoc> page = elasticsearchTemplate.queryForPage(query, DeviceRepairRecordDoc.class); //获得分组对象 InternalFilter repairPending = (InternalFilter) page.getAggregation("Pending"); InternalFilter repairProcessing = (InternalFilter) page.getAggregation("Processing"); //接收总数 DeviceRepairCountDTO deviceRepairCountDTO = new DeviceRepairCountDTO(); deviceRepairCountDTO.setPendingCount(repairPending.getDocCount()); deviceRepairCountDTO.setProcessCount(repairProcessing.getDocCount()); deviceRepairCountDTO.setCount(page.getTotalElements()); return deviceRepairCountDTO; }
3.3 重写分页规则
@Override public Page<ConsumptionBillDTO> getConsumptionListByAccount(Pageable pageable, Long accountId) { //重新书写分页顺序条件 Sort sort = pageable.getSort().and(Sort.by(Sort.Direction.DESC, "createdDate")); pageable = PageRequest.of(pageable.getPageNumber(), pageable.getPageSize(), sort); //BoolQueryBuilder查询必须匹配某个字段 BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery(); if (accountId != null) { queryBuilder.must(QueryBuilders.termQuery("accountId", accountId)); } SearchQuery query = new NativeSearchQueryBuilder() .withQuery(queryBuilder) .withPageable(pageable) .build(); //分页查询 Page<OrderIdxDoc> page = orderIdxRepository.search(query); List<OrderIdxDoc> list = page.getContent(); //list.stream().map 将一个OrderIdxDoc对象之后封装为另一个对象,简化遍历的操作 List<ConsumptionBillDTO> retList = list.stream().map(orderIdx -> orderMapper.to(orderIdx)).collect(Collectors.toList()); return new PageImpl(retList, pageable, page.getTotalElements()); }
3.4 去重总数和范围
/* * 获取设备消耗指标 * */ public DeviceConsumeMetric getDeviceConsumeMetric(long deviceId, DeviceQuery deviceQuery) { //布尔查询 BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery(); //匹配 deviceId后面的这个属性值是否和这个字段里面的值匹配 匹配到就过滤 queryBuilder.must(QueryBuilders.matchQuery("deviceId", deviceId)); //开始时间,继续放入搜索 if (deviceQuery.getStartDate() != null) { queryBuilder.must(QueryBuilders.rangeQuery("createdDate").gte(deviceQuery.getStartDate())); } //结束时间 if (deviceQuery.getEndDate() != null) { queryBuilder.must(QueryBuilders.rangeQuery("createdDate").lte(deviceQuery.getEndDate())); } //本地查询方法 SearchQuery searchQuery = new NativeSearchQueryBuilder() //将上述查询条件加入 .withQuery(queryBuilder) //聚合总数 .addAggregation(AggregationBuilders.sum("amount").field("amount")) .addAggregation(AggregationBuilders.cardinality("person").field("accountId")) .build(); //之后获得分组后的对象 AggregatedPage<ConsumeRecordDoc> terms = elasticsearchTemplate.queryForPage(searchQuery, ConsumeRecordDoc.class); //之后得到Sum类 Sum sumAmount = (InternalSum) terms.getAggregation("amount"); //去重 InternalCardinality terms1 = (InternalCardinality) terms.getAggregation("person"); DeviceConsumeMetric m = new DeviceConsumeMetric(); m.setAmount(BigDecimal.valueOf(sumAmount.getValue())); m.setUserCount(Math.toIntExact(terms1.getValue())); m.setCount(Math.toIntExact(terms.getTotalElements())); return m; }
3.5 提高阈值防止数据没有全部查到
public Page<DeviceConsumeInfo> getConsumeUserCountInfo(DeviceQuery deviceQuery, Pageable pageable) { int size = pageable.getPageSize() * ((pageable.getPageNumber()) + 1); int shard_size = (int) (size * 1.5) + 10; SearchQuery query = new NativeSearchQueryBuilder() .withQuery(getFilterdDevice(deviceQuery)) .addAggregation(AggregationBuilders.terms("deviceCount").size(size).shardSize(shard_size).field("deviceId") .subAggregation(AggregationBuilders.cardinality("userCount").field("accountId").precisionThreshold(40000)) .size(Integer.MAX_VALUE) ) .withPageable(pageable).withTrackScores(true) .build(); AggregatedPage<ConsumeRecordDoc> page = elasticsearchTemplate.queryForPage(query, ConsumeRecordDoc.class); Terms terms = (Terms) page.getAggregation("deviceCount"); long total = terms.getBuckets().size(); List<DeviceConsumeInfo> ret = terms.getBuckets().stream() .skip(pageable.getPageSize() * pageable.getPageNumber()) .limit(pageable.getPageSize()) .map(x -> { DeviceConsumeInfo info = new DeviceConsumeInfo(); long devId = (long) x.getKeyAsNumber(); InternalCardinality cardinality = x.getAggregations().get("userCount"); long userCount = cardinality.getValue(); info.setUserCount(userCount); info.setDeviceId(devId); return info; }) .collect(Collectors.toList()); return new PageImpl(ret, pageable, total); }
