04-ElasticSearch入门-阿里云开发者社区

一.安装 ElasticSearch

1.注意事项

注意事项：

内存不能太小，否则会启动失败

JDK 版本需要对应，es7 需要 Java 11

不能以 root 用户启动

### 2.安装 java11

#java版本查看

java-version

#下载安装

yuminstalljava-11-openjdk.x86_64-y

#查看位置

ls-rl$(whichjava)

#修改环境变量

vim/etc/profile

exportJAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.19.0.7-1.0.1.al8.x86_64

exportPATH=$PATH:$JAVA_HOME/bin

#使生效

source/etc/profile

3.下载 es

下载地址

#解压到到/usr/local/目录下

tar-zxvfelasticsearch-7.12.0-linux-x86_64.tar.gz-C/usr/local/

#进入解压后的目录

cd/usr/local/elasticsearch-7.12.0

#创建文件和日志目录(日志文件已经存在)

mkdirdata

修改配置:

vim config/elasticsearch.yml

cluster.name: my-application #集群名称

node.name: node-1 #节点名称

path.data: /usr/local/elasticsearch-7.12.0/data

path.logs: /usr/local/elasticsearch-7.12.0/logs

#设置绑定的ip，设置为0.0.0.0以后就可以让任何计算机节点访问到了

network.host: 0.0.0.0

http.port: 9200 #端口

#设置在集群中的所有节点名称，这个节点名称就是之前所修改的，目前是单机，放入一个节点即可

cluster.initial_master_nodes: ["node-1"]

4.增加用户

#创建用户

useradd elasticsearch

#用户加密码(elastic-search1)

passwd elasticsearch

5.赋权限

#修改目录权限至新增的elasticsearch用户

chown -R elasticsearch:elasticsearch /usr/local/elasticsearch-7.12.0

6.修改系统配置

修改系统中允许应用最多创建多少文件等的限制权限。Linux 默认来说，一般限制应用最多创建的文件是 65535 个。但是 ES 至少需要 65536 的文件创建权限。
修改系统中允许用户启动的进程开启多少个线程。默认的 Linux 限制 root 用户开启的进程可以开启任意数量的线程，其他用户开启的进程可以开启 1024 个线程。必须修改限制数为 4096+。因为 ES 至少需要 4096 的线程池预备。ES 在 5.x 版本之后，强制要求在 linux 中不能使用 root 用户启动 ES 进程。所以必须使用其他用户启动 ES 进程才可以。
Linux 低版本内核为线程分配的内存是 128K。4.x 版本的内核分配的内存更大。如果虚拟机的内存是 1G，最多只能开启 3000+个线程数。至少为虚拟机分配 1.5G 以上的内存。

#修改系统配置

vim /etc/security/limits.conf

# 追加到末尾即可

elasticsearch soft nofile 65536

elasticsearch hard nofile 65536

elasticsearch soft nproc 4096

elasticsearch hard nproc 4096

7.内存权限

vim /etc/sysctl.conf

vm.max_map_count=262144

#使配置生效

/sbin/sysctl -p

8.启动

#切换用户

su elasticsearch

#进入目录

cd /usr/local/elasticsearch-7.12.0

#启动

./bin/elasticsearch

./bin/elasticsearch -d

#启动Kibana

nohup sh /usr/local/kibana/bin/kibana &

9.验证

#查询端口信息

netstat -ntlp | grep -E '9200|5601'

#linux访问

curl 127.0.0.1:9200

#浏览器访问

http://47.119.162.180:9200

#查询集群状态

http://47.119.162.180:9200/cluster/health

解释：Status：集群状态。Green所有分片可用。Yellow所有主分片可用。Red主分片不可用，集群不可用。

{

"name": "node-1",

"cluster_name": "my-application",

"cluster_uuid": "jFwKZO8cT12BdPtU63m-ew",

"version": {

"number": "7.12.0",

"build_flavor": "default",

"build_type": "tar",

"build_hash": "78722783c38caa25a70982b5b042074cde5d3b3a",

"build_date": "2021-03-18T06:17:15.410153305Z",

"build_snapshot": false,

"lucene_version": "8.8.0",

"minimum_wire_compatibility_version": "6.8.0",

"minimum_index_compatibility_version": "6.0.0-beta1"

"tagline": "You Know, for Search"

}

二.安装 Kibana

1.下载 Kibana

下载地址

选择要下载的版本尽量和 ES 版本保持一致,版本采用 7.12.0 并选择 linux 版本

2.安装

#安装目录

cd /usr/local

#解压

tar -zxvf kibana-7.12.0-linux-x86_64.tar.gz

#改名

mv kibana-7.12.0-linux-x86_64 kibana

3.修改配置

修改 kibana.yml

cd /usr/local/kibana

vim config/kibana.yml

server.port: 5601 #kibana端口

server.host: "0.0.0.0" #所有主机都能访问，或者也可以指定一个ip

elasticsearch.hosts: "http://es服务公网IP:9200" #配置es的访问地址

kibana.index: ".kibana"

i18n.locale: "zh-CN" #配置项默认是英文,配置成中文的

#root

chown -R elasticsearch:elasticsearch /usr/local/kibana

4.启动

#切换用户

su elasticsearch

#启动

cd /usr/local/kibana/bin

./kabana

#后台启动

nohup sh /usr/local/kibana/bin/kibana &

5.验证

#端口号

netstat -ntlp | grep 5601

#浏览器

http://47.119.162.180:5601/

#控制台

http://47.119.162.180:5601/app/dev_tools#/console

6.测试

GET _search

{

"query": {

"match_all": {}

}

GET /

GET /_cluster/health

GET /_cat/health?v

GET /_cat/indices?v

PUT /demo_index?pretty

DELETE /demo_index?pretty

PUT /book

PUT /book/_doc/1

{

"id":1,

"title":"这是一文章",

"content":"xxxxx",

"comment":"备注信息",

"mobile":"13344556677"

}

PUT /book/_doc/2

{

"id":1,

"title":"这是一11文章",

"content":"xxxxx",

"comment":"备注信息",

"mobile":"13344556677"

}

GET /book/_doc/1

POST /book/_doc/1/_update

{

"doc": {

"title": "这是一333文章"

}

POST /book/_update/1

{

"doc": {

"title": "这是一3333333444555文章"

}

DELETE /book/_doc/1

POST /book/_doc/

{

"id":1,

"title":"这是一11文章",

"content":"xxxxx",

"comment":"备注信息",

"mobile":"13344556677"

}

GET /book/_doc/1

GET /book/_doc/1?_source_includes=id,title

PUT /read_index/_doc/1/_create

{

"id":1,

"title":"这是一11文章",

"content":"xxxxx",

"comment":"备注信息",

"mobile":"13344556677"

}

#插入数据

PUT /test_index/_doc/6

{

"num": 0

}

#执行脚本

POST /test_index/_doc/6/_update

{

"script": "ctx._source.num+=1"

}

#查询数据

GET /test_index/_doc/6

GET /test_index/_search

7.kibana 控制台

http://47.119.162.180:5601/app/dev_tools#/console

8.Grok 测试

9.格式化

10.历史记录

11.仪表板

三.简单使用

1.获取基本信息

get /

{

"name": "node-1",

"cluster_name": "my-application",

"cluster_uuid": "gmjBLVYEQFO6ZR2bveXFig",

"version": {

"number": "7.12.0",

"build_flavor": "default",

"build_type": "tar",

"build_hash": "78722783c38caa25a70982b5b042074cde5d3b3a",

"build_date": "2021-03-18T06:17:15.410153305Z",

"build_snapshot": false,

"lucene_version": "8.8.0",

"minimum_wire_compatibility_version": "6.8.0",

"minimum_index_compatibility_version": "6.0.0-beta1"

"tagline": "You Know, for Search"

}

2.集群状态

GET /_cluster/health

解释：Status：集群状态。Green 所有分片可用。Yellow 所有主分片可用。Red 主分片不可用，集群不可用。

{

"cluster_name": "my-application",

"status": "yellow",

"timed_out": false,

"number_of_nodes": 1,

"number_of_data_nodes": 1,

"active_primary_shards": 8,

"active_shards": 8,

"relocating_shards": 0,

"initializing_shards": 0,

"unassigned_shards": 1,

"delayed_unassigned_shards": 0,

"number_of_pending_tasks": 0,

"number_of_in_flight_fetch": 0,

"task_max_waiting_in_queue_millis": 0,

"active_shards_percent_as_number": 88.88888888888889

}

GET /_cat/health?v

green:每个索引的 primary shard 和 replica shard 都是 active 状态的.
yellow:每个索引的 primary shard 都是 active 状态的，但是部分 replica shard 不是 active 状态，处于不可用的状态
red：不是所有索引的 primary shard 都是 active 状态的，部分索引有数据丢失了

3.查看索引

GET /_cat/indices?v

4.创建索引

PUT /demo_index?pretty

{

"acknowledged": true,

"shards_acknowledged": true,

"index": "demo_index"

}

5.删除索引

DELETE /demo_index?pretty

{

"acknowledged": true

}

6.主键 id

手动 id

PUT /book/_doc/2

{

"id":1,

"title":"这是一11文章",

"content":"xxxxx",

"comment":"备注信息",

"mobile":"13344556677"

}

自动 id

POST /book/_doc/

{

"id":1,

"title":"这是一11文章",

"content":"xxxxx",

"comment":"备注信息",

"mobile":"13344556677"

}

自动id特点:长度为 20 个字符，URL 安全，base64 编码，GUID，分布式生成不冲突

{

"_index": "book",

"_type": "_doc",

"_id": "Qk9rAIgBMxAD_MReKtAQ",

"_version": 1,

"result": "created",

"_shards": {

"total": 2,

"successful": 1,

"failed": 0

"_seq_no": 8,

"_primary_term": 1

}

7.插入文档

put /blog_index/_doc/2

{

"id":1,

"title":"这是一篇文章",

"content":"xxxxx",

"comment":"备注信息",

"mobile":"13344556677"

}

{

"_index": "blog_index",

"_type": "_doc",

"_id": "2",

"_version": 1,

"result": "created",

"_shards": {

"total": 2,

"successful": 1,

"failed": 0

"_seq_no": 1,

"_primary_term": 1

}

8.查询文档

get /blog_index/_doc/1

{

"_index": "blog_index",

"_type": "_doc",

"_id": "1",

"_version": 1,

"_seq_no": 0,

"_primary_term": 1,

"found": true,

"_source": {

"id": 1,

"title": "这是一篇文章",

"content": "xxxxx",

"comment": "备注信息",

"mobile": "13344556677"

}

9.图书 crud

创建 book 索引

PUT /book

插入数据

PUT /book/_doc/1

{

"id":1,

"title":"这是一篇文章",

"content":"xxxxx",

"comment":"备注信息",

"mobile":"13344556677"

}

查询数据

GET /book/_doc/1

使用 put 全量替换

实质：日文档的内容不会立即删除，只是标记为 deleted。适当的时机，集群会将这些文档删除。

PUT /book/_doc/2

{

"id":1,

"title":"这是一11文章",

"content":"xxxxx",

"comment":"备注信息",

"mobile":"13344556677"

}

局部更新

es 内部获取旧文档
将传来的文档 field 更新到旧数据（内存）
将旧文档标记问 delete
创建新文档

POST /book/_doc/1/_update

{

"doc": {

"title": "这是一333文章"

}

noop:

多次执行"result" : "noop’
no operation

局部更新

POST /book/_update/1

{

"doc": {

"title": "这是一3333333444555文章"

}

删除数据

DELETE /book/_doc/1

四.高阶使用

1.属性分析

版本号
删除的时候做标记,使用的是懒删除
_id 分为手动 id 和默认 id

{

"_index": "book",

"_type": "_doc",

"_id": "1",

"_version": 6,

"_seq_no": 7,

"_primary_term": 1,

"found": true,

"_source": {

"id": 1,

"title": "这是一文章",

"content": "xxxxx",

"comment": "备注信息",

"mobile": "13344556677"

}

2._source 字段

含义：插入数据时的所有字段和值。在 get 获取数据时，在 source 字段中原样返回。

GET /book/_doc/1

定制返回:

GET /book/_doc/1?_source_includes=id,title

{

"_index" : "book",

"_type" : "_doc",

"_id" : "1",

"_version" : 6,

"_seq_no" : 7,

"_primary_term" : 1,

"found" : true,

"_source" : {

"id" : 1,

"title" : "这是一文章"

}

3.强制创建

为防止覆盖原有数据，我们在新增时，设置为强制创建，不会覆盖原有文档。

PUT /index/_doc/1/_create

PUT /read_index/_doc/1/_create

{

"id":1,

"title":"这是一11文章",

"content":"xxxxx",

"comment":"备注信息",

"mobile":"13344556677"

}

{

"_index": "read_index",

"_type": "_doc",

"_id": "1",

"_version": 1,

"result": "created",

"_shards": {

"total": 2,

"successful": 1,

"failed": 0

"_seq_no": 0,

"_primary_term": 1

}

4.脚本使用

#插入数据

PUT /test_index/_doc/6

{

"num": 0

}

#执行脚本

POST /test_index/_doc/6/_update

{

"script": "ctx._source.num+=1"

}

#查询数据

GET /test_index/_doc/6

{

"_index": "test_index",

"_type": "_doc",

"_id": "6",

"_version": 2,

"_seq_no": 1,

"_primary_term": 1,

"found": true,

"_source": {

"num": 1

}

5.查询索引

GET /test_index/_search

{

"took": 339,

"timed_out": false,

"_shards": {

"total": 1,

"successful": 1,

"skipped": 0,

"failed": 0

"hits": {

"total": {

"value": 1,

"relation": "eq"

"max_score": 1.0,

"hits": [

{

"_index": "test_index",

"_type": "_doc",

"_id": "6",

"_score": 1.0,

"_source": {

"num": 1

}

]

}

6._version 字段

删除的时候是异步删除,只是做了删除标记,延时删除策略.

es 内部主从同步时，是多线程异步，乐观锁机制。也是基于版本号的

线程 1 先到，线程 2 后到，副本数据是没有问题的
线程 2 先到，线程 1 后到：

副本分片先把数据改为 test3,verion=3.
线程 1 请求到了，副本分片，判断请求的 verison=1 太旧了，就会丢弃这个请求。

说明:

version 使用数据自带的 version 版本号
_version&version_type=external 则是并发时使用程序自己指定的 version,且是不存在的

PUT /test_index/_doc/4?version=2&version_type=external

{

"test_field": "itcast1"

}

7.重试

指定重试次数

POST /test_index/_doc/5/_update?retry_on_conflict=3

{

"doc": {

"test_field": "itcast1"

}

结合 version

POST /test_index/_doc/5/_update?retry_on_conflict=3&version=22&version_type=external

{

"doc": {

"test_field": "itcast1"

}

8.批量查询 mget

单条查询 GET /test_index/_doc/1，如果查询多个 id 的文档一条一条查询，网络开销太大。

GET /_mget

{

"docs" : [

{

"_index" : "test_index",

"_type" : "_doc",

"_id" : 1

{

"_index" : "test_index",

"_type" : "_doc",

"_id" : 7

}

]

}

{

"docs": [

{

"_index": "test_index",

"_type": "_doc",

"_id": "2",

"_version": 6,

"_seq_no": 12,

"_primary_term": 1,

"found": true,

"_source": {

"test_field": "test12333123321321"

}

{

"_index": "test_index",

"_type": "_doc",

"_id": "3",

"_version": 6,

"_seq_no": 18,

"_primary_term": 1,

"found": true,

"_source": {

"test_field": "test3213"

}

]

}

提示去掉 type

GET /_mget

{

"docs" : [

{

"_index" : "test_index",

"_id" : 2

{

"_index" : "test_index",

"_id" : 3

}

]

}

同一索引下批量查询：

GET /test_index/_mget

{

"docs" : [

{

"_id" : 2

{

"_id" : 3

}

]

}

第三种写法：搜索写法

post /test_index/_doc/_search

{

"query": {

"ids" : {

"values" : ["1", "7"]

}

9.bulk

Bulk 操作解释将文档的增删改查一些列操作，通过一次请求全都做完。减少网络传输次数。

#语法

POST /_bulk

{"action": {"metadata"}}

{"data"}

示例:

#如下操作，删除5，新增14，修改2。

POST /_bulk

{ "create": { "_index": "test_index","_id": "8"}}

{ "test_field": "test8" }

{ "update": { "_index": "test_index","_id": "3"} }

{ "doc": {"test_field": "bulk test"} }

{ "delete": { "_index": "test_index","_id": "5" }}

总结:

功能：

delete：删除一个文档，只要 1 个 json 串就可以了
create:相当于强制创建 PUT /index/type/id/_create
index：普通的 put 操作，可以是创建文档，也可以是全量替换文档
update：执行的是局部更新 partial update 操作

格式：每个 json 不能换行。相邻 json 必须换行。
隔离：每个操作互不影响。操作失败的行会返回其失败信息。
实际用法：bulk 请求一次不要太大，否则一下积压到内存中，性能会下降。所以，一次请求几千个操作、大小在几 M 正好。

10.查询节点

/_cat/nodes?v

五.分词器

1.mysql 热更新词库

1.下载源码

https://github.com/medcl/elasticsearch-analysis-ik/releases

ik 分词器，是个标准的 java maven 工程，直接导入 idea 就可以看到源码

2.修改源

org.wltea.analyzer.dic.Dictionary 类，160 行 Dictionary 单例类的初始化方法，在这里需要创建一个我们自定义的线程，并且启动它
org.wltea.analyzer.dic.HotDictReloadThread 类：就是死循环，不断调用 Dictionary.getSingleton().reLoadMainDict()，去重新加载词典
Dictionary 类，399 行：this.loadMySQLExtDict(); 加载 mymsql 字典。
Dictionary 类，609 行：this.loadMySQLStopwordDict();加载 mysql 停用词
config 下 jdbc-reload.properties。mysql 配置文件

3.mvn package 打包代码

target\releases\elasticsearch-analysis-ik-7.3.0.zip

4.解压缩 ik 压缩包

将 mysql 驱动 jar，放入 ik 的目录下

5.修改 jdbc 相关配置

6.重启 es

观察日志，日志中就会显示我们打印的那些东西，比如加载了什么配置，加载了什么词语，什么停用词

7.在 mysql 中添加词库与停用词

8.分词实验，验证热更新生效

GET /_analyze

{

"analyzer": "ik_smart",

"text": "喊麦"

}

2.ik 分词器种类

standard 分词器
ik_max_word 分词器
ik_smart 分词器

3.standard 分词器

GET /_analyze

{

"analyzer": "standard",

"text": "中华人民共和国人民大会堂"

}

{

"tokens": [

{

"token": "中",

"start_offset": 0,

"end_offset": 1,

"type": "<IDEOGRAPHIC>",

"position": 0

{

"token": "华",

"start_offset": 1,

"end_offset": 2,

"type": "<IDEOGRAPHIC>",

"position": 1

{

"token": "人",

"start_offset": 2,

"end_offset": 3,

"type": "<IDEOGRAPHIC>",

"position": 2

{

"token": "民",

"start_offset": 3,

"end_offset": 4,

"type": "<IDEOGRAPHIC>",

"position": 3

{

"token": "共",

"start_offset": 4,

"end_offset": 5,

"type": "<IDEOGRAPHIC>",

"position": 4

{

"token": "和",

"start_offset": 5,

"end_offset": 6,

"type": "<IDEOGRAPHIC>",

"position": 5

{

"token": "国",

"start_offset": 6,

"end_offset": 7,

"type": "<IDEOGRAPHIC>",

"position": 6

{

"token": "人",

"start_offset": 7,

"end_offset": 8,

"type": "<IDEOGRAPHIC>",

"position": 7

{

"token": "民",

"start_offset": 8,

"end_offset": 9,

"type": "<IDEOGRAPHIC>",

"position": 8

{

"token": "大",

"start_offset": 9,

"end_offset": 10,

"type": "<IDEOGRAPHIC>",

"position": 9

{

"token": "会",

"start_offset": 10,

"end_offset": 11,

"type": "<IDEOGRAPHIC>",

"position": 10

{

"token": "堂",

"start_offset": 11,

"end_offset": 12,

"type": "<IDEOGRAPHIC>",

"position": 11

}

]

}

4.ik_max_word

GET /_analyze

{

"analyzer": "ik_max_word",

"text": "中华人民共和国人民大会堂"

}

{

"tokens": [

{

"token": "中华人民共和国",

"start_offset": 0,

"end_offset": 7,

"type": "CN_WORD",

"position": 0

{

"token": "中华人民",

"start_offset": 0,

"end_offset": 4,

"type": "CN_WORD",

"position": 1

{

"token": "中华",

"start_offset": 0,

"end_offset": 2,

"type": "CN_WORD",

"position": 2

{

"token": "华人",

"start_offset": 1,

"end_offset": 3,

"type": "CN_WORD",

"position": 3

{

"token": "人民共和国",

"start_offset": 2,

"end_offset": 7,

"type": "CN_WORD",

"position": 4

{

"token": "人民",

"start_offset": 2,

"end_offset": 4,

"type": "CN_WORD",

"position": 5

{

"token": "共和国",

"start_offset": 4,

"end_offset": 7,

"type": "CN_WORD",

"position": 6

{

"token": "共和",

"start_offset": 4,

"end_offset": 6,

"type": "CN_WORD",

"position": 7

{

"token": "国人",

"start_offset": 6,

"end_offset": 8,

"type": "CN_WORD",

"position": 8

{

"token": "人民大会堂",

"start_offset": 7,

"end_offset": 12,

"type": "CN_WORD",

"position": 9

{

"token": "人民大会",

"start_offset": 7,

"end_offset": 11,

"type": "CN_WORD",

"position": 10

{

"token": "人民",

"start_offset": 7,

"end_offset": 9,

"type": "CN_WORD",

"position": 11

{

"token": "大会堂",

"start_offset": 9,

"end_offset": 12,

"type": "CN_WORD",

"position": 12

{

"token": "大会",

"start_offset": 9,

"end_offset": 11,

"type": "CN_WORD",

"position": 13

{

"token": "会堂",

"start_offset": 10,

"end_offset": 12,

"type": "CN_WORD",

"position": 14

}

]

}

5.ik_smart

GET /_analyze

{

"analyzer": "ik_smart",

"text": "中华人民共和国人民大会堂"

}

{

"tokens": [

{

"token": "中华人民共和国",

"start_offset": 0,

"end_offset": 7,

"type": "CN_WORD",

"position": 0

{

"token": "人民大会堂",

"start_offset": 7,

"end_offset": 12,

"type": "CN_WORD",

"position": 1

}

]

}

六.search 搜索

1.query string search

无条件搜索所有

GET /book/_search

{

"took": 969,

"timed_out": false,

"_shards": {

"total": 1,

"successful": 1,

"skipped": 0,

"failed": 0

"hits": {

"total": {

"value": 3,

"relation": "eq"

"max_score": 1.0,

"hits": [

{

"_index": "book",

"_type": "_doc",

"_id": "1",

"_score": 1.0,

"_source": {

"name": "Bootstrap开发",

"description": "Bootstrap是由Twitter推出的一个前台页面开发css框架，是一个非常流行的开发框架，此框架集成了多种页面效果。此开发框架包含了大量的CSS、JS程序代码，可以帮助开发者（尤其是不擅长css页面开发的程序人员）轻松的实现一个css，不受浏览器限制的精美界面css效果。",

"studymodel": "201002",

"price": 38.6,

"timestamp": "2019-08-25 19:11:35",

"pic": "group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",

"tags": ["bootstrap", "dev"]

}

]

}

2.带参数搜索

与 http 请求传参类似

GET /book/_search?q=name:java&sort=price:desc

类比 sql: select * from book where name like ’ %java%’ order by price desc

{

"took": 2,

"timed_out": false,

"_shards": {

"total": 1,

"successful": 1,

"skipped": 0,

"failed": 0

"hits": {

"total": {

"value": 1,

"relation": "eq"

"max_score": null,

"hits": [

{

"_index": "book",

"_type": "_doc",

"_id": "2",

"_score": null,

"_source": {

"name": "java编程思想",

"description": "java语言是世界第一编程语言，在软件开发领域使用人数最多。",

"studymodel": "201001",

"price": 68.6,

"timestamp": "2019-08-25 19:11:35",

"pic": "group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",

"tags": ["java", "dev"]

"sort": [68.6]

}

]

}

3. 设置 timeout

GET /book/_search?timeout=10ms

设置搜索的超时时间,到超时就返回

全局设置：配置文件中设置 search.default_search_timeout：100ms。默认不超时。

4.多索引搜索

multi-index 搜索模式

告诉你如何一次性搜索多个 index 和多个 type 下的数据

#所有索引下的所有数据都搜索出来

/_search

#指定一个index，搜索其下所有的数据

/index1/_search

#同时搜索两个index下的数据

/index1,index2/_search

#按照通配符去匹配多个索引

/index*/_search

5.分页搜索

分页搜索的语法:

select * from book limit 1,5

GET /book/_search?size=10

GET /book/_search?size=10&from=0

GET /book/_search?size=10&from=20

GET /book/_search?from=0&size=3

6.query string 基础语法

GET /book/_search?q=name:java
GET /book/_search?q=+name:java
GET /book/_search?q=-name:java

说明:

q=field:search content 的语法，就是根据字段进行搜索

+与没有+一样都是必须包含，-是必须不包含

7._all metadata 的原理和作用

不带字段的查询,会存在 all 索引中_,方便查询

GET /book/_search?q=java

直接可以搜索所有的 field，任意一个 field 包含指定的关键字就可以搜索出来。我们在进行中搜索的时候，难道是对 document 中的每一个 field 都进行一次搜索吗？不是的。

es 中_all 元数据。建立索引的时候，插入一条 docunment，es 会将所有的 field 值经行全量分词，把这些分词，放到_all field 中。在搜索的时候，没有指定 field，就在_all 搜索。

8.query DSL

查询全部 GET /book/_search

GET /book/_search

{

"query": { "match_all": {} }

}

排序 GET /book/_search?sort=price:desc

GET /book/_search

{

"query" : {

"match" : {

"name" : " java"

}

"sort": [

{ "price": "desc" }

]

}

分页查询 GET /book/_search?size=10&from=0

GET /book/_search

{

"query": { "match_all": {} },

"from": 0,

"size": 1

}

指定返回字段 GET /book/ _search? _source=name,studymodel

GET /book/_search

{

"query": { "match_all": {} },

"_source": ["name", "studymodel"]

}

通过组合以上各种类型查询，实现复杂查询。

9.组合多个搜索条件

搜索需求：title 必须包含 elasticsearch，content 可以包含 elasticsearch 也可以不包含，author_id 必须不为 111

sql where and or !=

初始数据：

POST /website/_doc/1

{

"title": "my hadoop article",

"content": "hadoop is very bad",

"author_id": 111

}

POST /website/_doc/2

{

"title": "my elasticsearch article",

"content": "es is very bad",

"author_id": 112

}

POST /website/_doc/3

{

"title": "my elasticsearch article",

"content": "es is very goods",

"author_id": 111

}

搜索：

GET /website/_doc/_search

{

"query": {

"bool": {

"must": [

{

"match": {

"title": "elasticsearch"

}

"should": [

{

"match": {

"content": "elasticsearch"

}

"must_not": [

{

"match": {

"author_id": 111

}

]

}

{

"took": 488,

"timed_out": false,

"_shards": {

"total": 1,

"successful": 1,

"skipped": 0,

"failed": 0

"hits": {

"total": {

"value": 1,

"relation": "eq"

"max_score": 0.47000363,

"hits": [

{

"_index": "website",

"_type": "_doc",

"_id": "2",

"_score": 0.47000363,

"_source": {

"title": "my elasticsearch article",

"content": "es is very bad",

"author_id": 112

}

]

}

更复杂的搜索需求：

select * from test_index where name='tom' or (hired =true and (personality ='good' and rude != true ))

GET /test_index/_search

{

"query": {

"bool": {

"must": { "match":{ "name": "tom" }},

"should": [

{ "match":{ "hired": true }},

{ "bool": {

"must":{ "match": { "personality": "good" }},

"must_not": { "match": { "rude": true }}

}}

"minimum_should_match": 1

}

10.full-text search 全文检索

重新创建 book 索引

PUT /book/

{

"settings": {

"number_of_shards": 1,

"number_of_replicas": 0

"mappings": {

"properties": {

"name":{

"type": "text",

"analyzer": "ik_max_word",

"search_analyzer": "ik_smart"

"description":{

"type": "text",

"analyzer": "ik_max_word",

"search_analyzer": "ik_smart"

"studymodel":{

"type": "keyword"

"price":{

"type": "double"

"timestamp": {

"type": "date",

"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"

"pic":{

"type":"text",

"index":false

}

插入数据

PUT /book/_doc/1

{

"name": "Bootstrap开发",

"studymodel": "201002",

"price":38.6,

"timestamp":"2019-08-25 19:11:35",

"pic":"group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",

"tags": [ "bootstrap", "dev"]

}

PUT /book/_doc/2

{

"name": "java编程思想",

"description": "java语言是世界第一编程语言，在软件开发领域使用人数最多。",

"studymodel": "201001",

"price":68.6,

"timestamp":"2019-08-25 19:11:35",

"pic":"group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",

"tags": [ "java", "dev"]

}

PUT /book/_doc/3

{

"name": "spring开发基础",

"description": "spring 在java领域非常流行，java程序员都在用。",

"studymodel": "201001",

"price":88.6,

"timestamp":"2019-08-24 19:11:35",

"pic":"group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",

"tags": [ "spring", "java"]

}

搜索

GET /book/_search

{

"query" : {

"match" : {

"description" : "java程序员"

}

11._score

{

"took": 1,

"timed_out": false,

"_shards": {

"total": 1,

"successful": 1,

"skipped": 0,

"failed": 0

"hits": {

"total": {

"value": 2,

"relation": "eq"

"max_score": 2.137549,

"hits": [

{

"_index": "book",

"_type": "_doc",

"_id": "3",

"_score": 2.137549,

"_source": {

"name": "spring开发基础",

"description": "spring 在java领域非常流行，java程序员都在用。",

"studymodel": "201001",

"price": 88.6,

"timestamp": "2019-08-24 19:11:35",

"pic": "group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",

"tags": ["spring", "java"]

}

{

"_index": "book",

"_type": "_doc",

"_id": "2",

"_score": 0.57961315,

"_source": {

"name": "java编程思想",

"description": "java语言是世界第一编程语言，在软件开发领域使用人数最多。",

"studymodel": "201001",

"price": 68.6,

"timestamp": "2019-08-25 19:11:35",

"pic": "group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",

"tags": ["java", "dev"]

}

]

}

结果分析:

建立索引时, description 字段 term 倒排索引

java 2,3
程序员 3

搜索时，直接找 description 中含有 java 的文档 2,3，并且 3 号文档含有两个 java 字段，一个程序员，所以得分高，排在前面。2 号文档含有一个 java，排在后面。

12.Filter

需求：用户查询 description 中有"java 程序员"，并且价格大于 80 小于 90 的数据。

GET /book/_search

{

"query": {

"bool": {

"must": [

{

"match": {

"description": "java程序员"

}

{

"range": {

"price": {

"gte": 80,

"lte": 90

}

]

}

使用 filter:

GET /book/_search

{

"query": {

"bool": {

"must": [

{

"match": {

"description": "java程序员"

}

"filter": {

"range": {

"price": {

"gte": 80,

"lte": 90

}

13.filter 与 query 对比

filter，仅仅只是按照搜索条件过滤出需要的数据而已，不计算任何相关度分数，对相关度没有任何影响。

query，会去计算每个 document 相对于搜索条件的相关度，并按照相关度进行排序。

应用场景:

一般来说，如果你是在进行搜索，需要将最匹配搜索条件的数据先返回，那么用 query 如果你只是要根据一些条件筛选出一部分数据，不关注其排序，那么用 filter

性能比较:

filter，不需要计算相关度分数，不需要按照相关度分数进行排序，同时还有内置的自动 cache 最常使用 filter 的数据
query，相反，要计算相关度分数，按照分数进行排序，而且无法 cache 结果

14.定位错误语法

验证错误语句：

GET /book/_validate/query?explain

{

"query": {

"mach": {

"description": "java程序员"

}

{

"valid": false,

"error": "org.elasticsearch.common.ParsingException: no [query] registered for [mach]"

}

正确

GET /book/_validate/query?explain

{

"query": {

"match": {

"description": "java程序员"

}

{

"_shards": {

"total": 1,

"successful": 1,

"failed": 0

"valid": true,

"explanations": [

{

"index": "book",

"valid": true,

"explanation": "description:java description:程序员"

}

]

}

一般用在那种特别复杂庞大的搜索下，比如你一下子写了上百行的搜索，这个时候可以先用 validate api 去验证一下，搜索是否合法。

合法以后，explain 就像 mysql 的执行计划，可以看到搜索的目标等信息。

15.默认排序规则

默认情况下，是按照_score 降序排序的

然而，某些情况下，可能没有有用的_score，比如说 filter

GET book/_search

{

"query": {

"bool": {

"must": [

{

"match": {

"description": "java程序员"

}

]

}

当然，也可以是 constant_score

16.定制排序规则

相当于 sql 中 order by ?sort=sprice:desc

GET /book/_search

{

"query": {

"constant_score": {

"filter" : {

"term" : {

"studymodel" : "201001"

}

"sort": [

{

"price": {

"order": "asc"

}

]

}

17.Text 字段排序问题

如果对一个 text field 进行排序，结果往往不准确，因为分词后是多个单词，再排序就不是我们想要的结果了。

通常解决方案是，将一个 text field 建立两次索引，一个分词，用来进行搜索；一个不分词，用来进行排序。

PUT /website

{

"mappings": {

"properties": {

"title": {

"type": "text",

"fields": {

"keyword": {

"type": "keyword"

}

"content": {

"type": "text"

"post_date": {

"type": "date"

"author_id": {

"type": "long"

}

插入数据

PUT /website/_doc/1

{

"title": "first article",

"content": "this is my second article",

"post_date": "2019-01-01",

"author_id": 110

}

PUT /website/_doc/2

{

"title": "second article",

"content": "this is my second article",

"post_date": "2019-01-01",

"author_id": 110

}

PUT /website/_doc/3

{

"title": "third article",

"content": "this is my third article",

"post_date": "2019-01-02",

"author_id": 110

}

搜索

GET /website/_search

{

"query": {

"match_all": {}

"sort": [

{

"title.keyword": {

"order": "desc"

}

]

}

18.Scroll 分批查询

场景：下载某一个索引中 1 亿条数据，到文件或是数据库。

不能一下全查出来，系统内存溢出。所以使用 scoll 滚动搜索技术，一批一批查询。

scoll 搜索会在第一次搜索的时候，保存一个当时的视图快照，之后只会基于该旧的视图快照提供数据搜索，如果这个期间数据变更，是不会让用户看到的

每次发送 scroll 请求，我们还需要指定一个 scoll 参数，指定一个时间窗口，每次搜索请求只要在这个时间窗口内能完成就可以了。

搜索

GET /book/_search?scroll=1m

{

"query": {

"match_all": {}

"size": 3

}

{

"_scroll_id": "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAMOkWTURBNDUtcjZTVUdKMFp5cXloVElOQQ==",

"took": 3,

"timed_out": false,

"_shards": {

"total": 1,

"successful": 1,

"skipped": 0,

"failed": 0

"hits": {

"total": {

"value": 3,

"relation": "eq"

"max_score": 1.0,

"hits": []

}

获得的结果会有一个 scoll_id，下一次再发送 scoll 请求的时候，必须带上这个 scoll_id

GET /_search/scroll

{

"scroll": "1m",

"scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAMOkWTURBNDUtcjZTVUdKMFp5cXloVElOQQ=="

}

与分页区别:

分页是给用户看的 deep paging
scroll 是用户系统内部操作，如下载批量数据，数据转移。零停机改变索引映射。

19.DSL 语法练习

1.match_all

GET /book/_search

{

"query": {

"match_all": {}

}

2.match

GET /book/_search

{

"query": {

"match": {

"description": "java程序员"

}

3.multi_match

GET /book/_search

{

"query": {

"multi_match": {

"query": "java程序员",

"fields": ["name", "description"]

}

4.range query 范围查询

GET /book/_search

{

"query": {

"range": {

"price": {

"gte": 80,

"lte": 90

}

5.term query

字段为 keyword 时，存储和搜索都不分词

GET /book/_search

{

"query": {

"term": {

"description": "java程序员"

}

6.terms query

GET /book/_search

{

"query": { "terms": { "tag": [ "search", "full_text", "nosql" ] }}

}

7.exist query

查询有某些字段值的文档

GET /_search

{

"query": {

"exists": {

"field": "join_date"

}

8.Fuzzy query

返回包含与搜索词类似的词的文档，该词由 Levenshtein 编辑距离度量。

包括以下几种情况：

更改角色（box→fox）
删除字符（aple→apple）
插入字符（sick→sic）
调换两个相邻字符（ACT→CAT）

GET /book/_search

{

"query": {

"fuzzy": {

"description": {

"value": "jave"

}

9.IDs

GET /book/_search

{

"query": {

"ids" : {

"values" : ["1", "4", "100"]

}

10.prefix 前缀查询

GET /book/_search

{

"query": {

"prefix": {

"description": {

"value": "spring"

}

11.regexp query

正则查询

GET /book/_search

{

"query": {

"regexp": {

"description": {

"value": "j.*a",

"flags" : "ALL",

"max_determinized_states": 10000,

"rewrite": "constant_score"

}

七.聚合示例

1.DSL API

1.all-分组-计数

计算每个 studymodel 下的商品数量

sql 语句： select studymodel，count(*) from book group by studymodel

GET /book/_search

{

"size": 0,

"query": {

"match_all": {}

"aggs": {

"group_by_model": {

"terms": { "field": "studymodel" }

}

2.分组-计数

计算每个 tags 下的商品数量

设置字段"fielddata": true,这个步骤很重要,不然会报错

在 es 中，text 类型的字段使用一种叫做 fielddata 的查询时内存数据结构。当字段被排序，聚合或者通过脚本访问时这种数据结构会被创建。它是通过从磁盘读取每个段的整个反向索引来构建的，然后存存储在 java 的堆内存中。

fileddata 默认是不开启的。Fielddata 可能会消耗大量的堆空间，尤其是在加载高基数文本字段时。一旦 fielddata 已加载到堆中，它将在该段的生命周期内保留。此外，加载 fielddata 是一个昂贵的过程，可能会导致用户遇到延迟命中。这就是默认情况下禁用 fielddata 的原因。如果尝试对文本字段进行排序，聚合或脚本访问，将看到以下异常：

“Fielddata is disabled on text fields by default. Set fielddata=true on [your_field_name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.”

在启用 fielddata 之前，请考虑使用文本字段进行聚合，排序或脚本的原因。这样做通常没有意义。text 字段在索引例如 New York 这样的词会被分词，会被拆成 new，york。在此字段上面来一个 terms 的聚合会返回一个 new 的 bucket 和一个 york 的 bucket，当你想只返回一个 New York 的 bucket 的时候就会出现问题。

PUT /book/_mapping/

{

"properties": {

"tags": {

"type": "text",

"fielddata": true

}

查询

GET /book/_search

{

"size": 0,

"query": {

"match_all": {}

"aggs": {

"group_by_tags": {

"terms": { "field": "tags" }

}

3.搜索-分组

加上搜索条件，计算每个 tags 下的商品数量

GET /book/_search

{

"size": 0,

"query": {

"match": {

"description": "java程序员"

}

"aggs": {

"group_by_tags": {

"terms": { "field": "tags" }

}

4.分组-平均

先分组，再算每组的平均值，计算每个 tag 下的商品的平均价格

GET /book/_search

{

"size": 0,

"aggs" : {

"group_by_tags" : {

"terms" : {

"field" : "tags"

"aggs" : {

"avg_price" : {

"avg" : { "field" : "price" }

}

5.分组-平均-排序

计算每个 tag 下的商品的平均价格，并且按照平均价格降序排序

GET /book/_search

{

"size": 0,

"aggs" : {

"group_by_tags" : {

"terms" : {

"field" : "tags",

"order": {

"avg_price": "desc"

}

"aggs" : {

"avg_price" : {

"avg" : { "field" : "price" }

}

6.分组-分组-组内平均

按照指定的价格范围区间进行分组，然后在每组内再按照 tag 进行分组，最后再计算每组的平均价格

GET /book/_search

{

"size": 0,

"aggs": {

"group_by_price": {

"range": {

"field": "price",

"ranges": [

{

"from": 0,

"to": 40

{

"from": 40,

"to": 60

{

"from": 60,

"to": 80

}

]

"aggs": {

"group_by_tags": {

"terms": {

"field": "tags"

"aggs": {

"average_price": {

"avg": {

"field": "price"

}

六.电视案例

创建索引及映射

PUT /tvs

PUT /tvs/_search

{

"properties": {

"price": {

"type": "long"

"color": {

"type": "keyword"

"brand": {

"type": "keyword"

"sold_date": {

"type": "date"

}

插入数据

POST /tvs/_bulk

{ "index": {}}

{ "price" : 1000, "color" : "红色", "brand" : "长虹", "sold_date" : "2019-10-28" }

{ "index": {}}

{ "price" : 2000, "color" : "红色", "brand" : "长虹", "sold_date" : "2019-11-05" }

{ "index": {}}

{ "price" : 3000, "color" : "绿色", "brand" : "小米", "sold_date" : "2019-05-18" }

{ "index": {}}

{ "price" : 1500, "color" : "蓝色", "brand" : "TCL", "sold_date" : "2019-07-02" }

{ "index": {}}

{ "price" : 1200, "color" : "绿色", "brand" : "TCL", "sold_date" : "2019-08-19" }

{ "index": {}}

{ "price" : 2000, "color" : "红色", "brand" : "长虹", "sold_date" : "2019-11-05" }

{ "index": {}}

{ "price" : 8000, "color" : "红色", "brand" : "三星", "sold_date" : "2020-01-01" }

{ "index": {}}

{ "price" : 2500, "color" : "蓝色", "brand" : "小米", "sold_date" : "2020-02-12" }

2.电视示例

1.统计哪种颜色的电视销量最高

GET /tvs/_search

{

"size" : 0,

"aggs" : {

"popular_colors" : {

"terms" : {

"field" : "color"

}

查询条件解析

size：只获取聚合结果，而不要执行聚合的原始数据aggs：固定语法，要对一份数据执行分组聚合操作popular_colors：就是对每个 aggs，都要起一个名字，terms：根据字段的值进行分组field：根据指定的字段的值进行分组

{

"took" : 18,

"timed_out" : false,

"_shards" : {

"total" : 1,

"successful" : 1,

"skipped" : 0,

"failed" : 0

"hits" : {

"total" : {

"value" : 8,

"relation" : "eq"

"max_score" : null,

"hits" : [ ]

"aggregations" : {

"popular_colors" : {

"doc_count_error_upper_bound" : 0,

"sum_other_doc_count" : 0,

"buckets" : [

{

"key" : "红色",

"doc_count" : 4

{

"key" : "绿色",

"doc_count" : 2

{

"key" : "蓝色",

"doc_count" : 2

}

]

}

返回结果解析

hits.hits：我们指定了 size 是 0，所以 hits.hits 就是空的aggregations：聚合结果popular_color：我们指定的某个聚合的名称buckets：根据我们指定的 field 划分出的 bucketskey：每个 bucket 对应的那个值doc_count：这个 bucket 分组内，有多少个数据数量，其实就是这种颜色的销量

每种颜色对应的 bucket 中的数据的默认的排序规则：按照 doc_count 降序排序

2,统计每种颜色电视平均价格

GET /tvs/_search

{

"size" : 0,

"aggs": {

"colors": {

"terms": {

"field": "color"

"aggs": {

"avg_price": {

"avg": {

"field": "price"

}

在一个 aggs 执行的 bucket 操作（terms），平级的 json 结构下，再加一个 aggs，这个第二个 aggs 内部，同样取个名字，执行一个 metric 操作，avg，对之前的每个 bucket 中的数据的指定的 field，price field，求一个平均值

{

"took": 4,

"timed_out": false,

"_shards": {

"total": 1,

"successful": 1,

"skipped": 0,

"failed": 0

"hits": {

"total": {

"value": 8,

"relation": "eq"

"max_score": null,

"hits": []

"aggregations": {

"colors": {

"doc_count_error_upper_bound": 0,

"sum_other_doc_count": 0,

"buckets": [

{

"key": "红色",

"doc_count": 4,

"avg_price": {

"value": 3250.0

}

{

"key": "绿色",

"doc_count": 2,

"avg_price": {

"value": 2100.0

}

{

"key": "蓝色",

"doc_count": 2,

"avg_price": {

"value": 2000.0

}

]

}

buckets，除了 key 和 doc_countavg_price：我们自己取的 metric aggs 的名字value：我们的 metric 计算的结果，每个 bucket 中的数据的 price 字段求平均值后的结果

相当于 sql: select avg(price) from tvs group by color

3.继续下钻分析

每个颜色下，平均价格及每个颜色下，每个品牌的平均价格

GET /tvs/_search

{

"size": 0,

"aggs": {

"group_by_color": {

"terms": {

"field": "color"

"aggs": {

"color_avg_price": {

"avg": {

"field": "price"

}

"group_by_brand": {

"terms": {

"field": "brand"

"aggs": {

"brand_avg_price": {

"avg": {

"field": "price"

}

4.更多的 metric

count：bucket，terms，自动就会有一个 doc_count，就相当于是 countavg：avg aggs，求平均值max：求一个 bucket 内，指定 field 值最大的那个数据min：求一个 bucket 内，指定 field 值最小的那个数据sum：求一个 bucket 内，指定 field 值的总和

GET /tvs/_search

{

"size" : 0,

"aggs": {

"colors": {

"terms": {

"field": "color"

"aggs": {

"avg_price": { "avg": { "field": "price" } },

"min_price" : { "min": { "field": "price"} },

"max_price" : { "max": { "field": "price"} },

"sum_price" : { "sum": { "field": "price" } }

}

5.划分范围 histogram

GET /tvs/_search

{

"size" : 0,

"aggs":{

"price":{

"histogram":{

"field": "price",

"interval": 2000

"aggs":{

"income": {

"sum": {

"field" : "price"

}

histogram：类似于 terms，也是进行 bucket 分组操作，接收一个 field，按照这个 field 的值的各个范围区间，进行 bucket 分组操作

"histogram":{

"field": "price",

"interval": 2000

}

interval：2000，划分范围，0_2000，20004000，4000_6000，60008000，8000~10000，buckets

bucket 有了之后，一样的，去对每个 bucket 执行 avg，count，sum，max，min，等各种 metric 操作，聚合分析

6.按照日期分组聚合

date_histogram，按照我们指定的某个 date 类型的日期 field，以及日期 interval，按照一定的日期间隔，去划分 bucket

min_doc_count：即使某个日期 interval，2017-01-01~2017-01-31 中，一条数据都没有，那么这个区间也是要返回的，不然默认是会过滤掉这个区间的extended_bounds，min，max：划分 bucket 的时候，会限定在这个起始日期，和截止日期内

GET /tvs/_search

{

"size" : 0,

"aggs": {

"sales": {

"date_histogram": {

"field": "sold_date",

"interval": "month",

"format": "yyyy-MM-dd",

"min_doc_count" : 0,

"extended_bounds" : {

"min" : "2019-01-01",

"max" : "2020-12-31"

}

7.统计每季度每个品牌的销售额

GET /tvs/_search

{

"size": 0,

"aggs": {

"group_by_sold_date": {

"date_histogram": {

"field": "sold_date",

"interval": "quarter",

"format": "yyyy-MM-dd",

"min_doc_count": 0,

"extended_bounds": {

"min": "2019-01-01",

"max": "2020-12-31"

}

"aggs": {

"group_by_brand": {

"terms": {

"field": "brand"

"aggs": {

"sum_price": {

"sum": {

"field": "price"

}

"total_sum_price": {

"sum": {

"field": "price"

}

8.搜索与聚合结合，查询某个品牌按颜色销量

搜索与聚合可以结合起来。

sql select count(*)

from tvs

where brand like "%小米%"

group by color

es aggregation，scope，任何的聚合，都必须在搜索出来的结果数据中之行，搜索结果，就是聚合分析操作的 scope

GET /tvs/_search

{

"size": 0,

"query": {

"term": {

"brand": {

"value": "小米"

}

"aggs": {

"group_by_color": {

"terms": {

"field": "color"

}

9.global bucket：单个品牌与所有品牌销量对比

aggregation，scope，一个聚合操作，必须在 query 的搜索结果范围内执行

出来两个结果，一个结果，是基于 query 搜索结果来聚合的; 一个结果，是对所有数据执行聚合的

GET /tvs/_search

{

"size": 0,

"query": {

"term": {

"brand": {

"value": "小米"

}

"aggs": {

"single_brand_avg_price": {

"avg": {

"field": "price"

}

"all": {

"global": {},

"aggs": {

"all_brand_avg_price": {

"avg": {

"field": "price"

}

10.过滤+聚合：统计价格大于 1200 的电视平均价格

搜索+聚合

过滤+聚合

GET /tvs/_search

{

"size": 0,

"query": {

"constant_score": {

"filter": {

"range": {

"price": {

"gte": 1200

}

"aggs": {

"avg_price": {

"avg": {

"field": "price"

}

11.bucket filter：统计品牌最近一个月的平均价格

GET /tvs/_search

{

"size": 0,

"query": {

"term": {

"brand": {

"value": "小米"

}

"aggs": {

"recent_150d": {

"filter": {

"range": {

"sold_date": {

"gte": "now-150d"

}

"aggs": {

"recent_150d_avg_price": {

"avg": {

"field": "price"

}

"recent_140d": {

"filter": {

"range": {

"sold_date": {

"gte": "now-140d"

}

"aggs": {

"recent_140d_avg_price": {

"avg": {

"field": "price"

}

"recent_130d": {

"filter": {

"range": {

"sold_date": {

"gte": "now-130d"

}

"aggs": {

"recent_130d_avg_price": {

"avg": {

"field": "price"

}

aggs.filter，针对的是聚合去做的

如果放 query 里面的 filter，是全局的，会对所有的数据都有影响

但是，如果，比如说，你要统计，长虹电视，最近 1 个月的平均值; 最近 3 个月的平均值; 最近 6 个月的平均值

bucket filter：对不同的 bucket 下的 aggs，进行 filter

12.按每种颜色的平均销售额降序排序

GET /tvs/_search

{

"size": 0,

"aggs": {

"group_by_color": {

"terms": {

"field": "color",

"order": {

"avg_price": "asc"

}

"aggs": {

"avg_price": {

"avg": {

"field": "price"

}

相当于 sql 子表数据字段可以立刻使用。

13.排序：按每种颜色的每种品牌平均销售额降序排序

GET /tvs/_search

{

"size": 0,

"aggs": {

"group_by_color": {

"terms": {

"field": "color"

"aggs": {

"group_by_brand": {

"terms": {

"field": "brand",

"order": {

"avg_price": "desc"

}

"aggs": {

"avg_price": {

"avg": {

"field": "price"

}

八.ELK 分析日志

1.日志分析

需求：集中收集分布式服务的日志

逻辑模块程序随时输出日志

package com.itheima.es;

import org.junit.Test;

import org.junit.runner.RunWith;

import org.slf4j.Logger;

import org.slf4j.LoggerFactory;

import org.springframework.boot.test.context.SpringBootTest;

import org.springframework.test.context.junit4.SpringRunner;

import java.util.Random;

/**

* creste by itheima.itcast

@SpringBootTest

@RunWith(SpringRunner.class)

public class TestLog {

private static final Logger LOGGER= LoggerFactory.getLogger(TestLog.class);

@Test

public void testLog(){

Random random =new Random();

while (true){

int userid=random.nextInt(10);

LOGGER.info("userId:{},send:{}",userid,"hello world.I am "+userid);

try {

Thread.sleep(500);

} catch (InterruptedException e) {

e.printStackTrace();

}

logstash 收集日志到 es
在 kibana 中查看日志,展示数据

2.grok 内置类型

写 logstash 配置文件

USERNAME [a-zA-Z0-9._-]+

USER %{USERNAME}

INT (?:[+-]?(?:[0-9]+))

BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))

NUMBER (?:%{BASE10NUM})

BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))

BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b

POSINT \b(?:[1-9][0-9]*)\b

NONNEGINT \b(?:[0-9]+)\b

WORD \b\w+\b

NOTSPACE \S+

SPACE \s*

DATA .*?

GREEDYDATA .*

QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))

UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}

# Networking

MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})

CISCOMAC (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})

WINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})

COMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})

IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?

IPV4 (?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])

IP (?:%{IPV6}|%{IPV4})

HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)

HOST %{HOSTNAME}

IPORHOST (?:%{HOSTNAME}|%{IP})

HOSTPORT %{IPORHOST}:%{POSINT}

# paths

PATH (?:%{UNIXPATH}|%{WINPATH})

UNIXPATH (?>/(?>[\w_%!$@:.,-]+|\\.)*)+

TTY (?:/dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+))

WINPATH (?>[A-Za-z]+:|\\)(?:\\[^\\?*]*)+

URIPROTO [A-Za-z]+(\+[A-Za-z+]+)?

URIHOST %{IPORHOST}(?::%{POSINT:port})?

# uripath comes loosely from RFC1738, but mostly from what Firefox

# doesn't turn into %XX

URIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+

#URIPARAM \?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?

URIPARAM \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]]*

URIPATHPARAM %{URIPATH}(?:%{URIPARAM})?

URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?

# Months: January, Feb, 3, 03, 12, December

MONTHNUM (?:0?[1-9]|1[0-2])

MONTHNUM2 (?:0[1-9]|1[0-2])

MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])

# Days: Monday, Tue, Thu, etc...

# Years?

YEAR (?>\d\d){1,2}

HOUR (?:2[0123]|[01]?[0-9])

MINUTE (?:[0-5][0-9])

# '60' is a leap second in most time standards and thus is valid.

SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)

TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])

# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)

DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}

DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}

ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))

ISO8601_SECOND (?:%{SECOND}|60)

TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?

DATE %{DATE_US}|%{DATE_EU}

DATESTAMP %{DATE}[- ]%{TIME}

TZ (?:[PMCE][SD]T|UTC)

DATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}

DATESTAMP_RFC2822 %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}

DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}

DATESTAMP_EVENTLOG %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}

# Syslog Dates: Month Day HH:MM:SS

SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}

PROG (?:[\w._/%-]+)

SYSLOGPROG %{PROG:program}(?:\[%{POSINT:pid}\])?

SYSLOGHOST %{IPORHOST}

SYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>

HTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}

# Shortcuts

QS %{QUOTEDSTRING}

# Log formats

SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:

COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

# Log Levels

九.Java API

代码地址

1.document

1.pom

<groupId>org.elasticsearch.client</groupId>

<artifactId>elasticsearch-rest-high-level-client</artifactId>

<groupId>org.elasticsearch</groupId>

<artifactId>elasticsearch</artifactId>

</exclusion>

</exclusions>

</dependency>

<groupId>org.elasticsearch</groupId>

<artifactId>elasticsearch</artifactId>

</dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter</artifactId>

<version>2.0.6.RELEASE</version>

</dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-test</artifactId>

<version>2.0.6.RELEASE</version>

</dependency>

<groupId>org.projectlombok</groupId>

<artifactId>lombok</artifactId>

</dependency>

</dependencies>

2.yaml

spring:

application:

name: search-service

kwan:

elasticsearch:

hostlist: 47.119.162.180:9200 #多个节点用逗号分隔

3.config

@Configuration

public class ElasticsearchConfig {

@Value("${kwan.elasticsearch.hostlist}")

private String hostlist;

@Bean(destroyMethod = "close")

public RestHighLevelClient restHighLevelClient() {

String[] split = hostlist.split(",");

HttpHost[] httpHostsArray = new HttpHost[split.length];

for (int i = 0; i < split.length; i++) {

String item = split[i];

httpHostsArray[i] = new HttpHost(item.split(":")[0], Integer.parseInt(item.split(":")[1]), "http");

}

return new RestHighLevelClient(RestClient.builder(httpHostsArray));

}

4.get

@Slf4j

@SpringBootTest(classes = SearchApplication.class)

@RunWith(SpringRunner.class)

public class TestDocument_01_get {

@Autowired

RestHighLevelClient client;

@Test

public void testGet() throws IOException {

//构建请求

GetRequest getRequest = new GetRequest("test_post", "1");

//添加可选参数

String[] includes = new String[]{"id", "comment"};

String[] excludes = Strings.EMPTY_ARRAY;

FetchSourceContext fetchSourceContext = new FetchSourceContext(true, includes, excludes);

getRequest.fetchSourceContext(fetchSourceContext);

//同步查询

GetResponse getResponse = client.get(getRequest, RequestOptions.DEFAULT);

//获取结果

if (getResponse.isExists()) {

log.info(getResponse.getId());

log.info(String.valueOf(getResponse.getVersion()));

log.info(getResponse.getSourceAsString());//以string获取数据

log.info(String.valueOf(getResponse.getSourceAsBytes()));////以Bytes获取数据

log.info(String.valueOf(getResponse.getSourceAsMap()));//以Map获取数据

} else {

log.info("数据不存在");

}

@Test

public void testGet() {

//构建请求

GetRequest getRequest = new GetRequest("test_post", "1");

//添加可选参数

String[] includes = new String[]{"id", "title"};

String[] excludes = Strings.EMPTY_ARRAY;

FetchSourceContext fetchSourceContext = new FetchSourceContext(true, includes, excludes);

getRequest.fetchSourceContext(fetchSourceContext);

//设置监听器

ActionListener<GetResponse> listener = new ActionListener<GetResponse>() {

//成功时

public void onResponse(GetResponse getResponse) {

log.info(getResponse.getId());

log.info(String.valueOf(getResponse.getVersion()));

log.info(getResponse.getSourceAsString());

}

//失败时

public void onFailure(Exception e) {

e.printStackTrace();

log.info("数据获取异常");

}

};

//异步查询

client.getAsync(getRequest, RequestOptions.DEFAULT, listener);

try {

Thread.sleep(5000);

} catch (InterruptedException e) {

e.printStackTrace();

}

5.add

@Test

public void testAdd() throws IOException {

//构建请求

IndexRequest request = new IndexRequest("test_post");

request.id("5");

//构建文档数据

String jsonString = "{\n" +

" \"user\":\"tomas\",\n" +

" \"postDate\":\"2019-07-18\",\n" +

" \"message\":\"trying out es1\"\n" +

"}";

request.source(jsonString, XContentType.JSON);

//同步

IndexResponse indexResponse = client.index(request, RequestOptions.DEFAULT);

//获取结果

log.info(indexResponse.getIndex());

log.info(indexResponse.getId());

log.info(String.valueOf(indexResponse.getResult()));

if (indexResponse.getResult() == DocWriteResponse.Result.CREATED) {

DocWriteResponse.Result result = indexResponse.getResult();

log.info("CREATE" + result);

} else if (indexResponse.getResult() == DocWriteResponse.Result.UPDATED) {

DocWriteResponse.Result result = indexResponse.getResult();

log.info("UPDATED" + result);

} else {

log.info("其他操作");

}

//获取分片信息

ReplicationResponse.ShardInfo shardInfo = indexResponse.getShardInfo();

if (shardInfo.getTotal() != shardInfo.getSuccessful()) {

log.info("处理成功的分片数少于总分片！");

}

if (shardInfo.getFailed() > 0) {

for (ReplicationResponse.ShardInfo.Failure failure : shardInfo.getFailures()) {

String reason = failure.reason();//每一个错误的原因

log.info(reason);

}

@Test

public void testAdd() throws IOException {

//构建请求

IndexRequest request = new IndexRequest("test_post");

request.id("6");

//构建文档数据

Map<String, Object> jsonMap = new HashMap<String, Object>();

jsonMap.put("user", "tomas");

jsonMap.put("postDate", "2019-07-18");

jsonMap.put("message", "trying out es1");

request.source(jsonMap);

//同步执行

IndexResponse indexResponse = client.index(request, RequestOptions.DEFAULT);

//获取结果

log.info(indexResponse.getIndex());

log.info(indexResponse.getId());

log.info(String.valueOf(indexResponse.getResult()));

if (indexResponse.getResult() == DocWriteResponse.Result.CREATED) {

DocWriteResponse.Result result = indexResponse.getResult();

log.info("CREATE" + result);

} else if (indexResponse.getResult() == DocWriteResponse.Result.UPDATED) {

DocWriteResponse.Result result = indexResponse.getResult();

log.info("UPDATED" + result);

}

ReplicationResponse.ShardInfo shardInfo = indexResponse.getShardInfo();

if (shardInfo.getTotal() != shardInfo.getSuccessful()) {

log.info("处理成功的分片数少于总分片！");

}

if (shardInfo.getFailed() > 0) {

for (ReplicationResponse.ShardInfo.Failure failure : shardInfo.getFailures()) {

String reason = failure.reason();//每一个错误的原因

log.info(reason);

}

@Test

public void testAdd() throws IOException {

//构建请求

IndexRequest request = new IndexRequest("test_post");

request.id("7");

//构建文档数据

XContentBuilder builder = XContentFactory.jsonBuilder();

builder.startObject();

{

builder.field("user", "tomas");

builder.field("message", "trying out es1");

builder.timeField("postDate", "2019-07-18");

}

builder.endObject();

request.source(builder);

//同步执行

IndexResponse indexResponse = client.index(request, RequestOptions.DEFAULT);

//获取结果

log.info(indexResponse.getIndex());

log.info(indexResponse.getId());

log.info(String.valueOf(indexResponse.getResult()));

if (indexResponse.getResult() == DocWriteResponse.Result.CREATED) {

DocWriteResponse.Result result = indexResponse.getResult();

log.info("CREATE" + result);

} else if (indexResponse.getResult() == DocWriteResponse.Result.UPDATED) {

DocWriteResponse.Result result = indexResponse.getResult();

log.info("UPDATED" + result);

}

ReplicationResponse.ShardInfo shardInfo = indexResponse.getShardInfo();

if (shardInfo.getTotal() != shardInfo.getSuccessful()) {

log.info("处理成功的分片数少于总分片！");

}

if (shardInfo.getFailed() > 0) {

for (ReplicationResponse.ShardInfo.Failure failure : shardInfo.getFailures()) {

String reason = failure.reason();//每一个错误的原因

log.info(reason);

}

@Test

public void testAdd() throws IOException {

//构建请求

IndexRequest request = new IndexRequest("test_post");

request.id("9");

//构建文档数据

request.source("user", "tomas",

"message", "trying out es1",

"postDate", "2019-07-18");

//同步执行

IndexResponse indexResponse = client.index(request, RequestOptions.DEFAULT);

//获取结果

log.info(indexResponse.getIndex());

log.info(indexResponse.getId());

log.info(String.valueOf(indexResponse.getResult()));

if (indexResponse.getResult() == DocWriteResponse.Result.CREATED) {

DocWriteResponse.Result result = indexResponse.getResult();

log.info("CREATE" + result);

} else if (indexResponse.getResult() == DocWriteResponse.Result.UPDATED) {

DocWriteResponse.Result result = indexResponse.getResult();

log.info("UPDATED" + result);

}

ReplicationResponse.ShardInfo shardInfo = indexResponse.getShardInfo();

if (shardInfo.getTotal() != shardInfo.getSuccessful()) {

log.info("处理成功的分片数少于总分片！");

}

if (shardInfo.getFailed() > 0) {

for (ReplicationResponse.ShardInfo.Failure failure : shardInfo.getFailures()) {

String reason = failure.reason();//每一个错误的原因

log.info(reason);

}

@Test

public void testAdd() throws IOException {

//构建请求

IndexRequest request = new IndexRequest("test_post");

request.id("10");

//构建文档数据

String jsonString = "{\n" +

" \"user\":\"tomas\",\n" +

" \"postDate\":\"2019-07-18\",\n" +

" \"message\":\"trying out es1\"\n" +

"}";

request.source(jsonString, XContentType.JSON);

//设置超时时间

request.timeout("1s");

request.timeout(TimeValue.timeValueSeconds(1));

//手动维护版本号

request.version(4);

request.versionType(VersionType.EXTERNAL);

//同步执行

IndexResponse indexResponse = client.index(request, RequestOptions.DEFAULT);

//获取结果

log.info(indexResponse.getIndex());

log.info(indexResponse.getId());

log.info(String.valueOf(indexResponse.getResult()));

if (indexResponse.getResult() == DocWriteResponse.Result.CREATED) {

DocWriteResponse.Result result = indexResponse.getResult();

log.info("CREATE" + result);

} else if (indexResponse.getResult() == DocWriteResponse.Result.UPDATED) {

DocWriteResponse.Result result = indexResponse.getResult();

log.info("UPDATED" + result);

}

ReplicationResponse.ShardInfo shardInfo = indexResponse.getShardInfo();

if (shardInfo.getTotal() != shardInfo.getSuccessful()) {

log.info("处理成功的分片数少于总分片！");

}

if (shardInfo.getFailed() > 0) {

for (ReplicationResponse.ShardInfo.Failure failure : shardInfo.getFailures()) {

String reason = failure.reason();//每一个错误的原因

log.info(reason);

}

6.update

@Test

public void testUpdate() throws IOException {

//创建请求

UpdateRequest request = new UpdateRequest("test_post", "5");

Map<String, Object> jsonMap = new HashMap<>();

jsonMap.put("user", "tomas Lee");

request.doc(jsonMap);

request.timeout("1s");

request.retryOnConflict(3);//重试次数

//同步执行

UpdateResponse updateResponse = client.update(request, RequestOptions.DEFAULT);

//获取结果

updateResponse.getId();

updateResponse.getIndex();

//判断结果

if (updateResponse.getResult() == DocWriteResponse.Result.CREATED) {

DocWriteResponse.Result result = updateResponse.getResult();

log.info("CREATED：" + result);

} else if (updateResponse.getResult() == DocWriteResponse.Result.UPDATED) {

DocWriteResponse.Result result = updateResponse.getResult();

log.info("UPDATED：" + result);

} else if (updateResponse.getResult() == DocWriteResponse.Result.DELETED) {

DocWriteResponse.Result result = updateResponse.getResult();

log.info("DELETED：" + result);

} else if (updateResponse.getResult() == DocWriteResponse.Result.NOOP) {

//没有操作

DocWriteResponse.Result result = updateResponse.getResult();

log.info("NOOP：" + result);

}

7.delete

@Test

public void testDelete() throws IOException {

//创建请求

DeleteRequest request = new DeleteRequest("test_post", "3");

//执行

DeleteResponse deleteResponse = client.delete(request, RequestOptions.DEFAULT);

//获取结果

deleteResponse.getId();

deleteResponse.getIndex();

DocWriteResponse.Result result = deleteResponse.getResult();

log.info(result.toString());

}

8.bulk

@Test

public void testBulk() throws IOException {

//创建请求

BulkRequest request = new BulkRequest();

request.add(new IndexRequest("post").id("1").source(XContentType.JSON, "field", "1"));

request.add(new IndexRequest("post").id("2").source(XContentType.JSON, "field", "2"));

request.add(new UpdateRequest("post", "1").doc(XContentType.JSON, "field", "3"));

request.add(new DeleteRequest("post").id("2"));

//执行

BulkResponse bulkResponse = client.bulk(request, RequestOptions.DEFAULT);

//获取结果

for (BulkItemResponse itemResponse : bulkResponse) {

DocWriteResponse response = itemResponse.getResponse();

switch (itemResponse.getOpType()) {

case INDEX:

IndexResponse indexResponse = (IndexResponse) response;

log.info("INDEX:" + indexResponse.getResult());

break;

case CREATE:

IndexResponse createResponse = (IndexResponse) response;

log.info("CREATE:" + createResponse.getResult());

break;

case UPDATE:

UpdateResponse updateResponse = (UpdateResponse) response;

log.info("UPDATE:" + updateResponse.getResult());

break;

case DELETE:

DeleteResponse deleteResponse = (DeleteResponse) response;

log.info("DELETE:" + deleteResponse.getResult());

break;

}

2.index

1.创建索引

@SpringBootTest

@RunWith(SpringRunner.class)

public class TestIndex_01_Create {

@Autowired

private RestHighLevelClient client;

//创建索引

@Test

public void testCreateIndex() throws IOException {

// PUT /my_index

// {

// "settings": {

// "number_of_shards": 1,

// "number_of_replicas": 1

// },

// "mappings": {

// "properties": {

// "field1":{

// "type": "text"

// },

// "field2":{

// "type": "text"

// }

// },

// "aliases": {

// "default_index": {}

// }

//创建索引对象

CreateIndexRequest createIndexRequest = new CreateIndexRequest("itheima_book");

//设置参数

createIndexRequest.settings(Settings.builder().put("number_of_shards", "1").put("number_of_replicas", "0"));

//指定映射1

createIndexRequest.mapping(" {\n" +

" \t\"properties\": {\n" +

" \"name\":{\n" +

" \"type\":\"keyword\"\n" +

" },\n" +

" \"description\": {\n" +

" \"type\": \"text\"\n" +

" },\n" +

" \"price\":{\n" +

" \"type\":\"long\"\n" +

" },\n" +

" \"pic\":{\n" +

" \"type\":\"text\",\n" +

" \"index\":false\n" +

" }\n" +

" \t}\n" +

"}", XContentType.JSON);

//设置别名

createIndexRequest.alias(new Alias("itheima_index_new"));

// 额外参数

//设置超时时间

createIndexRequest.setTimeout(TimeValue.timeValueMinutes(2));

//设置主节点超时时间

createIndexRequest.setMasterTimeout(TimeValue.timeValueMinutes(1));

//在创建索引API返回响应之前等待的活动分片副本的数量，以int形式表示

createIndexRequest.waitForActiveShards(ActiveShardCount.from(2));

createIndexRequest.waitForActiveShards(ActiveShardCount.DEFAULT);

//操作索引的客户端

IndicesClient indices = client.indices();

//执行创建索引库

CreateIndexResponse createIndexResponse = indices.create(createIndexRequest, RequestOptions.DEFAULT);

//得到响应

boolean acknowledged = createIndexResponse.isAcknowledged();

//得到响应指示是否在超时前为索引中的每个分片启动了所需数量的碎片副本

boolean shardsAcknowledged = createIndexResponse.isShardsAcknowledged();

System.out.println("acknowledged:" + acknowledged);

System.out.println("shardsAcknowledged:" + shardsAcknowledged);

}

@SpringBootTest

@RunWith(SpringRunner.class)

public class TestIndex_04_Async {

@Autowired

private RestHighLevelClient client;

//创建索引异步方式

@Test

public void testCreateIndexAsync() {

//创建索引对象

CreateIndexRequest createIndexRequest = new CreateIndexRequest("itheima_book");

//设置参数

createIndexRequest.settings(Settings.builder().put("number_of_shards", "1").put("number_of_replicas", "0"));

//指定映射1

createIndexRequest.mapping(" {\n" +

" \t\"properties\": {\n" +

" \"name\":{\n" +

" \"type\":\"keyword\"\n" +

" },\n" +

" \"description\": {\n" +

" \"type\": \"text\"\n" +

" },\n" +

" \"price\":{\n" +

" \"type\":\"long\"\n" +

" },\n" +

" \"pic\":{\n" +

" \"type\":\"text\",\n" +

" \"index\":false\n" +

" }\n" +

" \t}\n" +

"}", XContentType.JSON);

//设置别名

createIndexRequest.alias(new Alias("itheima_index_new"));

// 额外参数

//设置超时时间

createIndexRequest.setTimeout(TimeValue.timeValueMinutes(2));

//设置主节点超时时间

createIndexRequest.setMasterTimeout(TimeValue.timeValueMinutes(1));

//在创建索引API返回响应之前等待的活动分片副本的数量，以int形式表示

createIndexRequest.waitForActiveShards(ActiveShardCount.from(2));

createIndexRequest.waitForActiveShards(ActiveShardCount.DEFAULT);

//操作索引的客户端

IndicesClient indices = client.indices();

//执行创建索引库

ActionListener<CreateIndexResponse> listener = new ActionListener<CreateIndexResponse>() {

@Override

public void onResponse(CreateIndexResponse createIndexResponse) {

//得到响应（全部）

boolean acknowledged = createIndexResponse.isAcknowledged();

//得到响应指示是否在超时前为索引中的每个分片启动了所需数量的碎片副本

boolean shardsAcknowledged = createIndexResponse.isShardsAcknowledged();

System.out.println("acknowledged:" + acknowledged);

System.out.println("shardsAcknowledged:" + shardsAcknowledged);

}

@Override

public void onFailure(Exception e) {

e.printStackTrace();

}

};

client.indices().createAsync(createIndexRequest, RequestOptions.DEFAULT, listener);

try {

Thread.sleep(5000);

} catch (InterruptedException e) {

e.printStackTrace();

}

2.删除索引

@Test

public void testDeleteIndex() throws IOException {

//创建删除索引请求

DeleteIndexRequest deleteIndexRequest = new DeleteIndexRequest("itheima_book");

// 执行

AcknowledgedResponse delete = client.indices().delete(deleteIndexRequest, RequestOptions.DEFAULT);

//得到相应

boolean acknowledged = delete.isAcknowledged();

System.out.println("acknowledged:" + acknowledged);

}

3.是否存在索引

@Test

public void testExistIndex() throws IOException {

GetIndexRequest request = new GetIndexRequest("itheima_book");

//参数

request.local(false);//从主节点返回本地索引信息状态

request.humanReadable(true);//以适合人类的格式返回

request.includeDefaults(false);//是否返回每个索引的所有默认配置

boolean exists = client.indices().exists(request, RequestOptions.DEFAULT);

System.out.println("exists:" + exists);

}

4.关闭索引

@Test

public void testCloseIndex() throws IOException {

CloseIndexRequest request = new CloseIndexRequest("itheima_book");

AcknowledgedResponse close = client.indices().close(request, RequestOptions.DEFAULT);

boolean acknowledged = close.isAcknowledged();

System.out.println("acknowledged:" + acknowledged);

}

5.打开索引

@Test

public void testOpenIndex() throws IOException {

OpenIndexRequest request = new OpenIndexRequest("itheima_book");

OpenIndexResponse open = client.indices().open(request, RequestOptions.DEFAULT);

boolean acknowledged = open.isAcknowledged();

System.out.println("acknowledged:" + acknowledged);

}

3.search

1.all

@SpringBootTest

@RunWith(SpringRunner.class)

public class TestSearch_01_all {

@Autowired

RestHighLevelClient client;

@Test

public void testSearchAll() throws IOException {

// GET book/_search

// {

// "query": {

// "match_all": {}

// }

//1构建搜索请求

SearchRequest searchRequest = new SearchRequest("book");

SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

searchSourceBuilder.query(QueryBuilders.matchAllQuery());

//获取某些字段

searchSourceBuilder.fetchSource(new String[]{"name"}, new String[]{});

searchRequest.source(searchSourceBuilder);

//2执行搜索

SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

//3获取结果

SearchHits hits = searchResponse.getHits();

//数据数据

SearchHit[] searchHits = hits.getHits();

System.out.println("--------------------------");

for (SearchHit hit : searchHits) {

String id = hit.getId();

float score = hit.getScore();

Map<String, Object> sourceAsMap = hit.getSourceAsMap();

String name = (String) sourceAsMap.get("name");

String description = (String) sourceAsMap.get("description");

Double price = (Double) sourceAsMap.get("price");

System.out.println("id:" + id);

System.out.println("score:" + score);

System.out.println("name:" + name);

System.out.println("description:" + description);

System.out.println("price:" + price);

System.out.println("==========================");

}

2.page

@Test

public void testSearchPage() throws IOException {

// GET book/_search

// {

// "query": {

// "match_all": {}

// },

// "from": 0,

// "size": 2

// }

//1构建搜索请求

SearchRequest searchRequest = new SearchRequest("book");

SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

searchSourceBuilder.query(QueryBuilders.matchAllQuery());

//第几页

int page = 1;

//每页几个

int size = 2;

//下标计算

int from = (page - 1) * size;

searchSourceBuilder.from(from);

searchSourceBuilder.size(size);

searchRequest.source(searchSourceBuilder);

//2执行搜索

SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

//3获取结果

SearchHits hits = searchResponse.getHits();

//数据数据

SearchHit[] searchHits = hits.getHits();

System.out.println("--------------------------");

for (SearchHit hit : searchHits) {

String id = hit.getId();

float score = hit.getScore();

Map<String, Object> sourceAsMap = hit.getSourceAsMap();

String name = (String) sourceAsMap.get("name");

String description = (String) sourceAsMap.get("description");

Double price = (Double) sourceAsMap.get("price");

System.out.println("id:" + id);

System.out.println("name:" + name);

System.out.println("description:" + description);

System.out.println("price:" + price);

System.out.println("==========================");

}

3.ids

@Test

public void testSearchIds() throws IOException {

// GET /book/_search

// {

// "query": {

// "ids" : {

// "values" : ["1", "4", "100"]

// }

//1构建搜索请求

SearchRequest searchRequest = new SearchRequest("book");

SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

searchSourceBuilder.query(QueryBuilders.idsQuery().addIds("1", "4", "100"));

searchRequest.source(searchSourceBuilder);

//2执行搜索

SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

//3获取结果

SearchHits hits = searchResponse.getHits();

//数据数据

SearchHit[] searchHits = hits.getHits();

System.out.println("--------------------------");

for (SearchHit hit : searchHits) {

String id = hit.getId();

float score = hit.getScore();

Map<String, Object> sourceAsMap = hit.getSourceAsMap();

String name = (String) sourceAsMap.get("name");

String description = (String) sourceAsMap.get("description");

Double price = (Double) sourceAsMap.get("price");

System.out.println("id:" + id);

System.out.println("name:" + name);

System.out.println("description:" + description);

System.out.println("price:" + price);

System.out.println("==========================");

}

4.match

@Test

public void testSearchMatch() throws IOException {

// GET /book/_search

// {

// "query": {

// "match": {

// "description": "java程序员"

// }

//1构建搜索请求

SearchRequest searchRequest = new SearchRequest("book");

SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

searchSourceBuilder.query(QueryBuilders.matchQuery("description", "java程序员"));

searchRequest.source(searchSourceBuilder);

//2执行搜索

SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

//3获取结果

SearchHits hits = searchResponse.getHits();

//数据数据

SearchHit[] searchHits = hits.getHits();

System.out.println("--------------------------");

for (SearchHit hit : searchHits) {

String id = hit.getId();

float score = hit.getScore();

Map<String, Object> sourceAsMap = hit.getSourceAsMap();

String name = (String) sourceAsMap.get("name");

String description = (String) sourceAsMap.get("description");

Double price = (Double) sourceAsMap.get("price");

System.out.println("id:" + id);

System.out.println("name:" + name);

System.out.println("description:" + description);

System.out.println("price:" + price);

System.out.println("==========================");

}

5.term

@Test

public void testSearchTerm() throws IOException {

// GET /book/_search

// {

// "query": {

// "term": {

// "description": "java程序员"

// }

//1构建搜索请求

SearchRequest searchRequest = new SearchRequest("book");

SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

searchSourceBuilder.query(QueryBuilders.termQuery("description", "java程序员"));

searchRequest.source(searchSourceBuilder);

//2执行搜索

SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

//3获取结果

SearchHits hits = searchResponse.getHits();

//数据数据

SearchHit[] searchHits = hits.getHits();

System.out.println("--------------------------");

for (SearchHit hit : searchHits) {

String id = hit.getId();

float score = hit.getScore();

Map<String, Object> sourceAsMap = hit.getSourceAsMap();

String name = (String) sourceAsMap.get("name");

String description = (String) sourceAsMap.get("description");

Double price = (Double) sourceAsMap.get("price");

System.out.println("id:" + id);

System.out.println("name:" + name);

System.out.println("description:" + description);

System.out.println("price:" + price);

System.out.println("==========================");

}

6.multi_match

@Test

public void testSearchMultiMatch() throws IOException {

// GET /book/_search

// {

// "query": {

// "multi_match": {

// "query": "java程序员",

// "fields": ["name", "description"]

// }

//1构建搜索请求

SearchRequest searchRequest = new SearchRequest("book");

SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

searchSourceBuilder.query(QueryBuilders.multiMatchQuery("java程序员", "name", "description"));

searchRequest.source(searchSourceBuilder);

//2执行搜索

SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

//3获取结果

SearchHits hits = searchResponse.getHits();

//数据数据

SearchHit[] searchHits = hits.getHits();

System.out.println("--------------------------");

for (SearchHit hit : searchHits) {

String id = hit.getId();

float score = hit.getScore();

Map<String, Object> sourceAsMap = hit.getSourceAsMap();

String name = (String) sourceAsMap.get("name");

String description = (String) sourceAsMap.get("description");

Double price = (Double) sourceAsMap.get("price");

System.out.println("id:" + id);

System.out.println("name:" + name);

System.out.println("description:" + description);

System.out.println("price:" + price);

System.out.println("==========================");

}

7.bool

@Test

public void testSearchBool() throws IOException {

// GET /book/_search

// {

// "query": {

// "bool": {

// "must": [

// {

// "multi_match": {

// "query": "java程序员",

// "fields": ["name","description"]

// }

// ],

// "should": [

// {

// "match": {

// "studymodel": "201001"

// }

// ]

// }

//1构建搜索请求

SearchRequest searchRequest = new SearchRequest("book");

SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

//构建multiMatch请求

MultiMatchQueryBuilder multiMatchQueryBuilder = QueryBuilders.multiMatchQuery("java程序员", "name", "description");

//构建match请求

MatchQueryBuilder matchQueryBuilder = QueryBuilders.matchQuery("studymodel", "201001");

BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();

boolQueryBuilder.must(multiMatchQueryBuilder);

boolQueryBuilder.should(matchQueryBuilder);

searchSourceBuilder.query(boolQueryBuilder);

searchRequest.source(searchSourceBuilder);

//2执行搜索

SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

//3获取结果

SearchHits hits = searchResponse.getHits();

//数据数据

SearchHit[] searchHits = hits.getHits();

System.out.println("--------------------------");

for (SearchHit hit : searchHits) {

String id = hit.getId();

float score = hit.getScore();

Map<String, Object> sourceAsMap = hit.getSourceAsMap();

String name = (String) sourceAsMap.get("name");

String description = (String) sourceAsMap.get("description");

Double price = (Double) sourceAsMap.get("price");

System.out.println("id:" + id);

System.out.println("name:" + name);

System.out.println("description:" + description);

System.out.println("price:" + price);

System.out.println("==========================");

}

8.filter

@Test

public void testSearchFilter() throws IOException {

// GET /book/_search

// {

// "query": {

// "bool": {

// "must": [

// {

// "multi_match": {

// "query": "java程序员",

// "fields": ["name","description"]

// }

// ],

// "should": [

// {

// "match": {

// "studymodel": "201001"

// }

// ],

// "filter": {

// "range": {

// "price": {

// "gte": 50,

// "lte": 90

// }

//1构建搜索请求

SearchRequest searchRequest = new SearchRequest("book");

SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

//构建multiMatch请求

MultiMatchQueryBuilder multiMatchQueryBuilder = QueryBuilders.multiMatchQuery("java程序员", "name", "description");

//构建match请求

MatchQueryBuilder matchQueryBuilder = QueryBuilders.matchQuery("studymodel", "201001");

BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();

boolQueryBuilder.must(multiMatchQueryBuilder);

boolQueryBuilder.should(matchQueryBuilder);

boolQueryBuilder.filter(QueryBuilders.rangeQuery("price").gte(50).lte(90));

searchSourceBuilder.query(boolQueryBuilder);

searchRequest.source(searchSourceBuilder);

//2执行搜索

SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

//3获取结果

SearchHits hits = searchResponse.getHits();

//数据数据

SearchHit[] searchHits = hits.getHits();

System.out.println("--------------------------");

for (SearchHit hit : searchHits) {

String id = hit.getId();

float score = hit.getScore();

Map<String, Object> sourceAsMap = hit.getSourceAsMap();

String name = (String) sourceAsMap.get("name");

String description = (String) sourceAsMap.get("description");

Double price = (Double) sourceAsMap.get("price");

System.out.println("id:" + id);

System.out.println("name:" + name);

System.out.println("description:" + description);

System.out.println("price:" + price);

System.out.println("==========================");

}

9.sort

@Test

public void testSearchSort() throws IOException {

// GET /book/_search

// {

// "query": {

// "bool": {

// "must": [

// {

// "multi_match": {

// "query": "java程序员",

// "fields": ["name","description"]

// }

// ],

// "should": [

// {

// "match": {

// "studymodel": "201001"

// }

// ],

// "filter": {

// "range": {

// "price": {

// "gte": 50,

// "lte": 90

// }

// },

// "sort": [

// {

// "price": {

// "order": "asc"

// }

// ]

// }

//1构建搜索请求

SearchRequest searchRequest = new SearchRequest("book");

SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

//构建multiMatch请求

MultiMatchQueryBuilder multiMatchQueryBuilder = QueryBuilders.multiMatchQuery("java程序员", "name", "description");

//构建match请求

MatchQueryBuilder matchQueryBuilder = QueryBuilders.matchQuery("studymodel", "201001");

BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();

boolQueryBuilder.must(multiMatchQueryBuilder);

boolQueryBuilder.should(matchQueryBuilder);

boolQueryBuilder.filter(QueryBuilders.rangeQuery("price").gte(50).lte(90));

searchSourceBuilder.query(boolQueryBuilder);

//按照价格升序

searchSourceBuilder.sort("price", SortOrder.ASC);

searchRequest.source(searchSourceBuilder);

//2执行搜索

SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

//3获取结果

SearchHits hits = searchResponse.getHits();

//数据数据

SearchHit[] searchHits = hits.getHits();

System.out.println("--------------------------");

for (SearchHit hit : searchHits) {

String id = hit.getId();

float score = hit.getScore();

Map<String, Object> sourceAsMap = hit.getSourceAsMap();

String name = (String) sourceAsMap.get("name");

String description = (String) sourceAsMap.get("description");

Double price = (Double) sourceAsMap.get("price");

System.out.println("id:" + id);

System.out.println("name:" + name);

System.out.println("description:" + description);

System.out.println("price:" + price);

System.out.println("==========================");

}

4.sql 功能

前提 es 拥有白金版功能:

kibana 中管理-》许可管理开启白金版试用

导入依赖:

<groupId>org.elasticsearch.plugin</groupId>

</dependency>

<id>elastic.co</id>

<url>https://artifacts.elastic.co/maven</url>

</repository>

</repositories>

代码:

public class TestJdbc {

public static void main(String[] args) {

//1创建连接

try {

Connection connection = DriverManager.getConnection("jdbc:es://http://localhost:9200");

//2创建statement

Statement statement = connection.createStatement();

//3执行sql语句

ResultSet resultSet = statement.executeQuery("SELECT * FROM tvs");

//4获取结果

while (resultSet.next()) {

System.out.println(resultSet.getString(1));

System.out.println(resultSet.getString(2));

System.out.println(resultSet.getString(3));

System.out.println(resultSet.getString(4));

System.out.println("======================================");

}

} catch (SQLException e) {

e.printStackTrace();

}

5.学成在线站内搜索模块

1.mysql 导入学生数据

Navicat Premium Data Transfer

Source Server : local

Source Server Type : MySQL

Source Server Version : 50721

Source Host : localhost:3306

Source Schema : xc_course

Target Server Type : MySQL

Target Server Version : 50721

File Encoding : 65001

Date: 10/11/2019 02:50:34

SET

NAMES utf8mb4;

SET

FOREIGN_KEY_CHECKS = 0;

-- ----------------------------

-- Table structure for course_pub

-- ----------------------------

DROP TABLE IF EXISTS `course_pub`;

CREATE TABLE `course_pub` (

`id` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '主键',

`name` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '课程名称',

`users` varchar(500) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '适用人群',

`mt` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '大分类',

`st` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '小分类',

`grade` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '课程等级',

`studymodel` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '学习模式',

`teachmode` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '教育模式',

`description` text CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '课程介绍',

`timestamp` timestamp(0) NOT NULL DEFAULT CURRENT_TIMESTAMP(0) COMMENT '时间戳logstash使用',

`charge` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '收费规则，对应数据字典',

`valid` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '有效性，对应数据字典',

`qq` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '咨询qq',

`price` float(10, 2) NULL DEFAULT NULL COMMENT '价格',

`price_old` float(10, 2) NULL DEFAULT NULL COMMENT '原价格',

`expires` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '过期时间',

`start_time` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '课程有效期-开始时间',

`end_time` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '课程有效期-结束时间',

`pic` varchar(500) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '课程图片',

`teachplan` text CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '课程计划',

`pub_time` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '发布时间',

PRIMARY KEY (`id`) USING BTREE

) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------

-- Records of course_pub

-- ----------------------------

INSERT INTO

`course_pub`

VALUES

(

'297e7c7c62b888f00162b8a7dec20000',

'test_java基础33',

'b1',

'1-3',

'1-3-3',

'200002',

'201002',

NULL,

'java 从入门到删库跑路',

'2019-10-28 11:26:25',

'203002',

'204002',

'32432',

NULL,

'group1/M00/00/00/wKgZhV2tIgiAaYVMAAA2T52Dthw246.jpg',

'{\"children\":[{\"children\":[],\"id\":\"40288f9b6e0c10d8016e0c37f72a0000\",\"pname\":\"1\"},{\"children\":[{\"id\":\"40288581632b593e01632bd53ff10001\",\"mediaFileoriginalname\":\"solr.avi\",\"mediaId\":\"5fbb79a2016c0eb609ecd0cd3dc48016\",\"pname\":\"Hello World\"},{\"id\":\"40288f9b6e106273016e106485f30000\",\"mediaFileoriginalname\":\"lucene.avi\",\"mediaId\":\"c5c75d70f382e6016d2f506d134eee11\",\"pname\":\"java基础\"}],\"id\":\"40288581632b593e01632bd4ec360000\",\"pname\":\"程序入门\"},{\"children\":[{\"id\":\"40288f9b6dce18e3016dcef16d860001\",\"mediaFileoriginalname\":\"solr.avi\",\"mediaId\":\"5fbb79a2016c0eb609ecd0cd3dc48016\",\"pname\":\"三级节点\"}],\"id\":\"40288f9b6dce18e3016dcef12a1d0000\",\"pname\":\"二级节点\"},{\"children\":[{\"id\":\"40288c9a6ca3968e016ca417fa8d0001\",\"mediaFileoriginalname\":\"lucene.avi\",\"mediaId\":\"c5c75d70f382e6016d2f506d134eee11\",\"pname\":\"test04-01\"}],\"id\":\"40288c9a6ca3968e016ca417b4a50000\",\"pname\":\"test04\"},{\"children\":[{\"id\":\"40288581632b593e01632bd5d31f0003\",\"mediaFileoriginalname\":\"solr.avi\",\"mediaId\":\"5fbb79a2016c0eb609ecd0cd3dc48016\",\"pname\":\"表达式\"},{\"id\":\"40288581632b593e01632bd606480004\",\"pname\":\"逻辑运算\"}],\"id\":\"40288581632b593e01632bd597810002\",\"pname\":\"编程基础\"},{\"children\":[{\"id\":\"402881e764034e4301640351f3d70003\",\"pname\":\"一切皆为对象\"}],\"id\":\"402881e764034e430164035091a00002\",\"pname\":\"面向对象\"},{\"children\":[{\"id\":\"402899816ad8457c016ad9282a330001\",\"pname\":\"test06\"}],\"id\":\"402899816ad8457c016ad927ba540000\",\"pname\":\"test05\"}],\"id\":\"4028858162bec7f30162becad8590000\",\"pname\":\"test_java基础33\"}',

'2019-10-28 11:26:24'

);

INSERT INTO

`course_pub`

VALUES

(

'297e7c7c62b888f00162b8a965510001',

'test_java基础node',

'test_java基础',

'1-3',

'1-3-2',

'200001',

'201001',

NULL,

'test_java基础2test_java基础2test_java基础2test_java基础2test_java基础2test_java基础2test_java基础2test_java基础2test_java基础2test_java基础2',

'2019-10-24 16:26:34',

'203001',

'204001',

'443242',

NULL,

'{\"children\":[{\"children\":[{\"id\":\"402881e66417407b01641744fc650001\",\"pname\":\"入门程序\"}],\"id\":\"402881e66417407b01641744afc30000\",\"pname\":\"基础知识\"},{\"children\":[],\"id\":\"4028858162e5d6e00162e5e0727d0001\",\"pname\":\"java基础语法\"},{\"children\":[{\"id\":\"4028d0866b158241016b502433d60002\",\"pname\":\"第二节\"}],\"id\":\"4028d0866b158241016b5023f51e0001\",\"pname\":\"第二章\"}],\"id\":\"4028858162e5d6e00162e5e0227b0000\",\"pname\":\"test_java基础2\"}',

'2019-10-24 16:26:33'

);

SET

FOREIGN_KEY_CHECKS = 1;

2.创建索引 xc_course

3.创建映射

PUT /xc_course

{

"settings": {

"number_of_shards": 1,

"number_of_replicas": 0

"mappings": {

"properties": {

"description" : {

"analyzer" : "ik_max_word",

"search_analyzer": "ik_smart",

"type" : "text"

"grade" : {

"type" : "keyword"

"id" : {

"type" : "keyword"

"mt" : {

"type" : "keyword"

"name" : {

"analyzer" : "ik_max_word",

"search_analyzer": "ik_smart",

"type" : "text"

"users" : {

"index" : false,

"type" : "text"

"charge" : {

"type" : "keyword"

"valid" : {

"type" : "keyword"

"pic" : {

"index" : false,

"type" : "keyword"

"qq" : {

"index" : false,

"type" : "keyword"

"price" : {

"type" : "float"

"price_old" : {

"type" : "float"

"st" : {

"type" : "keyword"

"status" : {

"type" : "keyword"

"studymodel" : {

"type" : "keyword"

"teachmode" : {

"type" : "keyword"

"teachplan" : {

"analyzer" : "ik_max_word",

"search_analyzer": "ik_smart",

"type" : "text"

"expires" : {

"type" : "date",

"format": "yyyy-MM-dd HH:mm:ss"

"pub_time" : {

"type" : "date",

"format": "yyyy-MM-dd HH:mm:ss"

"start_time" : {

"type" : "date",

"format": "yyyy-MM-dd HH:mm:ss"

"end_time" : {

"type" : "date",

"format": "yyyy-MM-dd HH:mm:ss"

}

4.logstash 创建模板文件

Logstash 的工作是从 MySQL 中读取数据，向 ES 中创建索引，这里需要提前创建 mapping 的模板文件以便 logstash 使用。

在 logstach 的 config 目录创建 xc_course_template.json，内容如下：

{

"mappings": {

"doc": {

"properties": {

"charge": {

"type": "keyword"

"description": {

"analyzer": "ik_max_word",

"search_analyzer": "ik_smart",

"type": "text"

"end_time": {

"format": "yyyy-MM-dd HH:mm:ss",

"type": "date"

"expires": {

"format": "yyyy-MM-dd HH:mm:ss",

"type": "date"

"grade": {

"type": "keyword"

"id": {

"type": "keyword"

"mt": {

"type": "keyword"

"name": {

"analyzer": "ik_max_word",

"search_analyzer": "ik_smart",

"type": "text"

"pic": {

"index": false,

"type": "keyword"

"price": {

"type": "float"

"price_old": {

"type": "float"

"pub_time": {

"format": "yyyy-MM-dd HH:mm:ss",

"type": "date"

"qq": {

"index": false,

"type": "keyword"

"st": {

"type": "keyword"

"start_time": {

"format": "yyyy-MM-dd HH:mm:ss",

"type": "date"

"status": {

"type": "keyword"

"studymodel": {

"type": "keyword"

"teachmode": {

"type": "keyword"

"teachplan": {

"analyzer": "ik_max_word",

"search_analyzer": "ik_smart",

"type": "text"

"users": {

"index": false,

"type": "text"

"valid": {

"type": "keyword"

}

"template": "xc_course"

}

5.logstash 配置 mysql.conf

ES 采用 UTC 时区问题

ES 采用 UTC 时区，比北京时间早 8 小时，所以 ES 读取数据时让最后更新时间加 8 小时

where timestamp > date_add(:sql_last_value,INTERVAL 8 HOUR)

logstash 每个执行完成会在/config/logstash_metadata 记录执行时间下次以此时间为基准进行增量同步数据到索引库。

6.启动

.\logstash.bat -f ..\config\mysql.conf

7.后端代码

代码地址

7.1.Controller

@RestController

@RequestMapping("/search/course")

public class EsCourseController {

@Autowired

private EsCourseServiceImpl esCourseServiceImpl;

@GetMapping(value = "/list/{page}/{size}")

public QueryResponseResult<CoursePub> list(@PathVariable("page") int page

, @PathVariable("size") int size, CourseSearchParam courseSearchParam) {

return esCourseServiceImpl.list(page, size, courseSearchParam);

}

7.2,service

@Service

publicclassEsCourseServiceImplimplementsEsCourseService {

@Value("${kwan.course.source_field}")

privateStringsource_field;

@Autowired

privateRestHighLevelClientrestHighLevelClient;

/**

* 课程搜索

* @param page

* @param size

* @param courseSearchParam

* @return

@Override

publicQueryResponseResult<CoursePub>list(intpage, intsize, CourseSearchParamcourseSearchParam) {

if (courseSearchParam==null) {

courseSearchParam=newCourseSearchParam();

}

//1创建搜索请求对象

SearchRequestsearchRequest=newSearchRequest("xc_course");

SearchSourceBuildersearchSourceBuilder=newSearchSourceBuilder();

//过虑源字段

String[] source_field_array=source_field.split(",");

searchSourceBuilder.fetchSource(source_field_array, newString[]{});

//创建布尔查询对象

BoolQueryBuilderboolQueryBuilder=QueryBuilders.boolQuery();

//搜索条件

//根据关键字搜索

if (StringUtils.isNotEmpty(courseSearchParam.getKeyword())) {

MultiMatchQueryBuildermultiMatchQueryBuilder=

QueryBuilders

.multiMatchQuery(courseSearchParam.getKeyword(), "name", "description", "teachplan")

.minimumShouldMatch("70%")

.field("name", 10);

boolQueryBuilder.must(multiMatchQueryBuilder);

}

if (StringUtils.isNotEmpty(courseSearchParam.getMt())) {

//根据一级分类

boolQueryBuilder.filter(QueryBuilders.termQuery("mt", courseSearchParam.getMt()));

}

if (StringUtils.isNotEmpty(courseSearchParam.getSt())) {

//根据二级分类

boolQueryBuilder.filter(QueryBuilders.termQuery("st", courseSearchParam.getSt()));

}

if (StringUtils.isNotEmpty(courseSearchParam.getGrade())) {

//根据难度等级

boolQueryBuilder.filter(QueryBuilders.termQuery("grade", courseSearchParam.getGrade()));

}

//设置boolQueryBuilder到searchSourceBuilder

searchSourceBuilder.query(boolQueryBuilder);

//设置分页参数

if (page<=0) {

page=1;

}

if (size<=0) {

size=12;

}

//起始记录下标

intfrom= (page-1) *size;

searchSourceBuilder.from(from);

searchSourceBuilder.size(size);

//设置高亮

HighlightBuilderhighlightBuilder=newHighlightBuilder();

highlightBuilder.preTags("<font class='eslight'>");

highlightBuilder.postTags("</font>");

//设置高亮字段

highlightBuilder.fields().add(newHighlightBuilder.Field("name"));

searchSourceBuilder.highlighter(highlightBuilder);

searchRequest.source(searchSourceBuilder);

QueryResult<CoursePub>queryResult=newQueryResult();

List<CoursePub>list=newArrayList<>();

try {

//2执行搜索

SearchResponsesearchResponse=restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);

//3获取响应结果

SearchHitshits=searchResponse.getHits();

longtotalHits=hits.getTotalHits().value;

//匹配的总记录数

queryResult.setTotal(totalHits);

SearchHit[] searchHits=hits.getHits();

for (SearchHithit : searchHits) {

CoursePubcoursePub=newCoursePub();

//源文档

Map<String, Object>sourceAsMap=hit.getSourceAsMap();

//取出id

Stringid= (String) sourceAsMap.get("id");

coursePub.setId(id);

//取出name

Stringname= (String) sourceAsMap.get("name");

//取出高亮字段name

Map<String, HighlightField>highlightFields=hit.getHighlightFields();

if (highlightFields!=null) {

HighlightFieldhighlightFieldName=highlightFields.get("name");

if (highlightFieldName!=null) {

Text[] fragments=highlightFieldName.fragments();

StringBufferstringBuffer=newStringBuffer();

for (Texttext : fragments) {

stringBuffer.append(text);

}

name=stringBuffer.toString();

}

coursePub.setName(name);

//图片

Stringpic= (String) sourceAsMap.get("pic");

coursePub.setPic(pic);

//价格

Doubleprice=null;

try {

if (sourceAsMap.get("price") !=null) {

price= (Double) sourceAsMap.get("price");

}

} catch (Exceptione) {

e.printStackTrace();

}

coursePub.setPrice(price);

//旧价格

Doubleprice_old=null;

try {

if (sourceAsMap.get("price_old") !=null) {

price_old= (Double) sourceAsMap.get("price_old");

}

} catch (Exceptione) {

e.printStackTrace();

}

coursePub.setPrice_old(price_old);

//将coursePub对象放入list

list.add(coursePub);

}

} catch (IOExceptione) {

e.printStackTrace();

}

queryResult.setList(list);

QueryResponseResult<CoursePub>queryResponseResult=newQueryResponseResult<>(CommonCode.SUCCESS, queryResult);

returnqueryResponseResult;

}

热门

活动广场

任务中心

开发者评测

高校计划

乘风者计划

训练营

阿里云MVP

话题

直播

下载

镜像站

技术资料

插件

04-ElasticSearch入门

一.安装 ElasticSearch

1.注意事项

3.下载 es

4.增加用户

5.赋权限

6.修改系统配置

7.内存权限

8.启动

9.验证

二.安装 Kibana

1.下载 Kibana

2.安装

3.修改配置

4.启动

5.验证

6.测试

7.kibana 控制台

8.Grok 测试

9.格式化

10.历史记录

11.仪表板

三.简单使用

1.获取基本信息

2.集群状态

3.查看索引

4.创建索引

5.删除索引

6.主键 id

7.插入文档

8.查询文档

9.图书 crud

四.高阶使用

1.属性分析

2._source 字段

3.强制创建

4.脚本使用

5.查询索引

6._version 字段

7.重试

8.批量查询 mget

9.bulk

10.查询节点

五.分词器

1.mysql 热更新词库

2.ik 分词器种类

3.standard 分词器

4.ik_max_word

5.ik_smart

六.search 搜索

1.query string search

2.带参数搜索

3. 设置 timeout

4.多索引搜索

5.分页搜索

6.query string 基础语法

7._all metadata 的原理和作用

8.query DSL

9.组合多个搜索条件

10.full-text search 全文检索

11._score

12.Filter

13.filter 与 query 对比

14.定位错误语法

15.默认排序规则

16.定制排序规则