ElasticSearch-2.0.0集群安装配置与API使用实践

本文涉及的产品
检索分析服务 Elasticsearch 版,2核4GB开发者规格 1个月
简介:

ElasticSearch是基于全文搜索引擎库Lucene构建的分布式搜索引擎,我们可以直接使用ElasticSearch实现分布式搜索系统的搭建与使用,都知道,Lucene只是一个搜索框架,它提供了搜索引擎操作的基本API,如果要实现一个能够使用的搜索引擎系统,还需要自己基于Lucene的API去实现,工作量很大,而且还需要很好地掌握Lucene的底层实现原理。
ElasticSearch是一个完整的分布式搜索引擎系统,它的一些基本特性包括如下:

  • 全文检索
  • 提供插件机制,可以共享重用插件的功能
  • 分布式文件存储
  • 分布式实时索引和搜索
  • 实时统计分析
  • 可以横向扩展,支持大规模数据的搜索
  • 简单易用的RESTful API
  • 基于Replication实现了数据的高可用特性
  • 与其他系统的集成
  • 支持结构化和非结构化数据
  • 灵活的Schema设计(Mappings)
  • 支持多编程语言客户端

我个人感觉,ElasticSearch尽量屏蔽底层Lucene相关的技术细节,让你根本无从感觉底层Lucene相关的内容,这样你可以省去了了解Lucene 的成本,学习曲线比较平缓,不像Solr,如果想要构造负责的查询(Query),还是要对Lucene有所了解的。另外,在分布式设计方面,ElasticSearch更轻量一些,用起来更简单,而使用Solr的分布式分片功能需要使用SolrCloud,它基于ZooKeeper来实现配置管理,以及Replication功能,而且Solr需要使用Web容器来部署,相对来说有点复杂一些(我个人之前使用的SolrCloud版本大概是3.1~3.5左右,比较早,现在可能更加完善了)。

基本概念

我们熟悉一下ElasticSearch中涉及到的一些基本概念:

  • 索引(Index)

索引(Index)是文档的集合,它是根据实际业务逻辑进行划分的,通常会把相对独立且具有相似结构或者性质的数据作为文档,放在一起,形成一个索引,比如,用户相关信息可以作为一个索引,交易相关信息也可应作为另一个索引。

  • 类型(Type)

类型(Type)是索引内部的一个逻辑划分,在一个索引内部可以定义多个类型(Type),类型将一个索引在逻辑上划分为多个集合,每个类型包含多个属性(字段)。比如,我们基于手机客户端应用App,创建一个了用户相关信息的索引,然后再在这个索引内部定义多个类型:基本信息类型、设备信息类型、行为信息类型,基本信息类型中包含用户编号、证件号码、名称、手机号码、年龄、出生日期,设备信息类型包括设备类型、设备名称、App版本号、渠道来源、系统版本、IMEI、mac地址,用户行为信息包含用户编号、事件编号、事件类型、时间、浏览页面代码、地区编码,这样有3个类型在一个索引当中。ElasticSearch中类型,与HBase中列簇(Column Family)的概念很相似。

  • 文档(Document)

文档(Document)是索引的基本单元,它与关系数据库中的一条记录相类似,包含了一组属性信息,同时包含一个唯一标识这一组属性值的ID,通过该ID可以更新一个文档,也可以删除一个文档。

  • 分片(Shards)&副本(Replicas)

一个索引是很多文档的集合,将一个索引进行分割,分成多个片段(一个索引的子集),每一个片段称为一个分片(Shard),这样划分可以很好地管理索引,跨节点存储,为分布式存储于搜索提供了便利。副本(Replica)是为了保证一个分片(Shard)的可用性,冗余复制存储,当一个分片对应的数据无法读取时,可以读取其副本,正常提供搜索服务。

集群安装配置

ElasticSearch集群安装配置非常容易,安装可以执行如下命令行:

1 wgethttps://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/zip/elasticsearch/2.0.0/elasticsearch-2.0.0.zip
2 unzip elasticsearch-2.0.0.zip

拿出集群的一个节点的进行配置,修改配置文件config/elasticsearch.yml的内容,如下所示:

01 # ======================== Elasticsearch Configuration =========================
02 #
03 # NOTE: Elasticsearch comes with reasonable defaults for most settings.
04 # Before you set out to tweak and tune the configuration, make sure you
05 # understand what are you trying to accomplish and the consequences.
06 #
07 # The primary way of configuring a node is via this file. This template lists
08 # the most important settings you may want to configure for a production cluster.
09 #
10 # Please see the documentation for further information on configuration options:
11 # <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
12 #
13 # ---------------------------------- Cluster -----------------------------------
14 #
15 # Use a descriptive name for your cluster:
16 #
17 cluster.name: dw_search_engine
18 #
19 # ------------------------------------ Node ------------------------------------
20 #
21 # Use a descriptive name for the node:
22 #
23 node.name: esnode-01
24 #
25 # Add custom attributes to the node:
26 #
27 # node.rack: r1
28 #
29 # ----------------------------------- Paths ------------------------------------
30 #
31 # Path to directory where to store the data (separate multiple locations by comma):
32 #
33 path.data: /data/dw_search_storage
34 #
35 # Path to log files:
36 #
37 path.logs: /tmp/es/logs
38 #
39 # ----------------------------------- Memory -----------------------------------
40 #
41 # Lock the memory on startup:
42 #
43 # bootstrap.mlockall: true
44 #
45 # Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
46 # available on the system and that the owner of the process is allowed to use this limit.
47 #
48 # Elasticsearch performs poorly when the system is swapping the memory.
49 #
50 # ---------------------------------- Network -----------------------------------
51 #
52 # Set the bind adress to a specific IP (IPv4 or IPv6):
53 #
54 network.host: 10.10.2.62
55 #
56 # Set a custom port for HTTP:
57 #
58 http.port: 9200
59 #
60 # For more information, see the documentation at:
61 # <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
62 #
63 # ---------------------------------- Gateway -----------------------------------
64 #
65 # Block initial recovery after a full cluster restart until N nodes are started:
66 #
67 # gateway.recover_after_nodes: 3
68 #
69 # For more information, see the documentation at:
70 # <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
71 #
72 # --------------------------------- Discovery ----------------------------------
73 #
74 # Elasticsearch nodes will find each other via unicast, by default.
75 #
76 # Pass an initial list of hosts to perform discovery when new node is started:
77 # The default list of hosts is ["127.0.0.1", "[::1]"]
78 #
79 discovery.zen.ping.unicast.hosts: ["es-01", "es-02"]
80 #
81 # Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
82 #
83 # discovery.zen.minimum_master_nodes: 3
84 #
85 # For more information, see the documentation at:
86 # <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
87 #
88 # ---------------------------------- Various -----------------------------------
89 #
90 # Disable starting multiple nodes on a single system:
91 #
92 # node.max_local_storage_nodes: 1
93 #
94 # Require explicit names when deleting indices:
95 #
96 # action.destructive_requires_name: true

其它节点的配置,在保证基本存储目录相同的前提下,可以根据需要修改如下几个参数:

1 node.name
2 network.host
3 http.port

最后,在每个节点上分别启动ElasticSearch,执行如下命令:

1 cd elasticsearch-2.0.0
2 bin/elasticsearch -d

然后可以查看Web管理界面,需要安装插件elasticsearch-head,后面会介绍,Web管理界面,如下所示:

0433694fa175e1c9dc442aea4773bb7993a2d2bc

上图中,我们已经创建了一个索引,可以看到节点的状态,及其分片(Shard)的情况。

RESTful API基本操作

尤其是在进行搜索的时候,为了使得其他系统能够与ElasticSearch搜索系统很好地解耦合,使用ElasticSearch提供的RESTful API是一种不错的选择。下面,我们介绍RESTful API的基本操作。

  • 插件管理

插件的存放目录为elasticsearch-2.0.0/plugins/,插件都是基于该存储目录进行操作的。
安装插件:

1 bin/plugin install analysis-icu
2 bin/plugin install mobz/elasticsearch-head

可以从不同的位置安装插件,上面第一个称为Core Elasticsearch plugin,它是Elasticsearch提供的,会从Elasticsearch上下载并安装;上面第一个是从Github上自动下载安装。还有其他的方式安装,如从特定的文件系统等进行安装。
列出插件:

1 bin/plugin list

删除插件:

1 bin/plugin remove analysis-icu

安装完一个插件,我们可以查看,例如查看elasticsearch_head插件,查看如下链接:

1 http://10.10.2.62:9200/_plugin/head/
  • 创建索引
1 curl -XPUT 'http://10.10.2.62:9200/basis_device_info/'

创建的索引名称为basis_device_info,我们也可以不指定一个索引对应的Mappings,而是在索引的时候自动生成Mappings,所以如果没有指定一个索引的Mappings,则这个索引可以支持任何的Mappings。同样可知,一个索引可以自动地增加不同的type,非常灵活。
也可以指定索引的基本配置,如分片(Shard)数目、副本(Replica)数目,如下所示:

1 curl -XPUT 'http://10.10.2.62:9200/basis_device_info /' -d '{
2 "settings" : {
3 "index" : {
4 "number_of_shards" : 10,
5 "number_of_replicas" : 1
6 }
7 }
8 }'

默认是5个分片,不进行复制,上面配置表示索引basis_device_info有10个分片,每个分片1个副本。
下面在创建索引的时候,指定设计的schema,即配置mappings,如下所示:

01 curl -XPUT 'http://10.10.2.62:9200/basis_device_info/' -d '
02 {
03 "mappings": {
04 "user": {
05 "_all": { "enabled": false },
06 "properties": {
07 "installid": { "type": "string" },
08 "appid": { "type": "string" },
09 "channel": { "type": "string", "index": "analyzed" },
10 "version": { "type": "string" },
11 "osversion": { "type": "string" },
12 "device_name": { "type": "string", "index": "analyzed" },
13 "producer": { "type": "string" },
14 "device_type": { "type": "string" },
15 "resolution": { "type": "string", "index": "analyzed" },
16 "screen_size": { "type": "string", "index": "analyzed" },
17 "mac": { "type": "string", "index": "not_analyzed" },
18 "idfa": { "type": "string" },
19 "idfv": { "type": "string", "index": "not_analyzed" },
20 "imei": { "type": "string", "index": "not_analyzed" },
21 "create_time": {
22 "type": "date",
23 "format": "yyyy-MM-dd HH:mm:ss",
24 "index": "not_analyzed"
25 }
26 }
27 }
28 }
29 }'

上面创建了索引basis_device_info,同时type为user,有了mappings,我们就知道需要索引的数据的格式了。

  • 删除索引
1 curl -XDELETE 'http://10.10.2.62:9200/basis_device_info/'

删除索引basis_device_info。

  • 索引文档
01 curl -PUT'http://10.10.2.62:9200/basis_device_info/user/CC49E748588490D41BFB89584007B0FA' -d '{
02 "installid": "0000000L",
03 "appid": "0",
04 "udid": "CC49E748588490D41BFB89584007B0FA",
05 "channel": "wulei1",
06 "version": "3.1.2",
07 "osversion": "8.1",
08 "device_name": "iPhone Retina4 Simulator",
09 "producer": "apple",
10 "device_type": "1",
11 "resolution": "640*1136",
12 "screen_size": "320*568",
13 "mac": "600308A20C5E",
14 "idfa": "dbbbs-fdsfa-fafda-321saf",
15 "idfv": "4283FAE1-19EB-4FA9-B739-8148F76BC8C3",
16 "imei": "af-sfd0fdsa-fad-ff",
17 "create_time": "2015-01-14 20:32:05"
18 }'

基于我们前面创建的type为user的索引,索引一个文档,文档_id为CC49E748588490D41BFB89584007B0FA,文档内容为一个用户设备信息,使用JSON格式表示。

  • 批量索引

批量索引,可以根据自己熟悉的编程语言或者脚本来实现,ElasticSearch也提供了一些客户端库。下面我们首先根据数据文件,构造成ElasticSearch索引支持的JSON格式,导出文件,然后通过curl工具去进行批量索引,实际上使用的是ElasticSearch提供的bulk API来实现的。
首先处理原始带索引数据,代码如下所示:

01 package org.shirdrn.es;
02
03 import java.io.BufferedReader;
04 import java.io.BufferedWriter;
05 import java.io.Closeable;
06 import java.io.File;
07 import java.io.FileReader;
08 import java.io.FileWriter;
09
10 import net.sf.json.JSONObject;
11
12 import com.google.common.base.Throwables;
13
14 public class EsIndexingClient {
15
16 public static void closeQuietly(Closeable... closeables) {
17 if(closeables != null) {
18 for(Closeable closeable : closeables) {
19 try {
20 closeable.close();
21 } catch (Exception e) { }
22 }
23 }
24 }
25
26 public static void main(String[] args) {
27 String f = "C:\\Users\\yanjun\\Desktop\\basis_device_info.txt";
28 String out = "C:\\Users\\yanjun\\Desktop\\basis_device_info.json";
29 File in = new File(f);
30 BufferedReader reader = null;
31 BufferedWriter writer = null;
32 try {
33 writer = new BufferedWriter(new FileWriter(out));
34 reader = new BufferedReader(new FileReader(in.getAbsoluteFile()));
35 String line = null;
36 while((line = reader.readLine()) != null) {
37 String[] a = line.split("\t", -1);
38 if(a.length == 16) {
39 String udid = a[2];
40
41 JSONObject c = new JSONObject();
42 c.put("_index", "basis_device_info");
43 c.put("_type", "user");
44 c.put("_id", udid);
45
46 JSONObject index = new JSONObject();
47 index.put("index", c);
48
49 JSONObject doc = new JSONObject();
50 doc.put("installid", a[0]);
51 doc.put("appid", a[1]);
52 doc.put("udid", a[2]);
53 doc.put("channel", a[3]);
54 doc.put("version", a[4]);
55 doc.put("osversion", a[5]);
56 doc.put("device_name", a[6]);
57 doc.put("producer", a[7]);
58 doc.put("device_type", a[8]);
59 doc.put("resolution", a[9]);
60 doc.put("screen_size", a[10]);
61 doc.put("mac", a[11]);
62 doc.put("idfa", a[12]);
63 doc.put("idfv", a[13]);
64 doc.put("imei", a[14]);
65 doc.put("create_time", a[15]);
66
67 writer.write(index.toString() + "\n");
68 writer.write(doc.toString() + "\n");
69 }
70 }
71
72 } catch (Exception e) {
73 throw Throwables.propagate(e);
74 } finally {
75 closeQuietly(reader, writer);
76 }
77
78 }
79 }

运行代码,输出的数据文件为basis_device_info.json,该文件的格式了,示例如下所示:

1 {"index":{"_index":"basis_device_info","_type":"user","_id":"1c207122a4b2c9632212ab86bac10f60"}}
2 {"installid":"00000002","appid":"0","udid":"1c207122a4b2c9632212ab86bac10f60","channel":"itings","version":"3.1.1","osversion":"4.1.2","device_name":"Lenovo P770","producer":"Lenovo","device_type":"0","resolution":"540*960","screen_size":"4.59","mac":"d4:22:3f:83:17:06","idfa":"","idfv":"","imei":"861166023335745","create_time":"2015-01-14 19:39:35"}
3 {"index":{"_index":"basis_device_info","_type":"user","_id":"FA6B1B98E6FF4E6994A1505A996F6102"}}
4 {"installid":"00000003","appid":"0","udid":"FA6B1B98E6FF4E6994A1505A996F6102","channel":"appstore","version":"3.1.1","osversion":"8.1.2","device_name":"iPhone 6Plus","producer":"apple","device_type":"1","resolution":"640*1136","screen_size":"320*568","mac":"020000000000","idfa":"84018625-A3C9-47A8-88D0-C57C12F80520","idfv":"9D1E2514-9DC8-47A8-ABD0-129FC0FB3171","imei":"","create_time":"2015-01-14 19:41:21"}
5 {"index":{"_index":"basis_device_info","_type":"user","_id":"8c5fe70b2408f184abcbe4f34b8f23c3"}}
6 {"installid":"00000004","appid":"0","udid":"8c5fe70b2408f184abcbe4f34b8f23c3","channel":"itings","version":"3.1.1.014","osversion":"4.2.2","device_name":"2014011","producer":"Xiaomi","device_type":"0","resolution":"720*1280","screen_size":"4.59","mac":"0c:1d:af:4f:48:9f","idfa":"","idfv":"","imei":"865763025472173","create_time":"2015-01-14 19:46:37"}

奇数编号行的内容为索引的指令信息,包括索引名称(_index)、类型(_type)、唯一标识(_id),偶数编号行的内容为实际待索引的文档数据。
然后,通过curl命令来进行批量索引,执行如下命令:

1 curl -s -XPOST http://10.10.2.62:9200/basis_device_info/_bulk --data-binary"@basis_device_info.json"
  • 搜索文档

简单的搜索,可以通过GET方式搜索,如下所示:

1 http://10.10.2.62:9200/basis_device_info/user/CC49E748588490D41BFB89584007B0FA
2
3
4 http://10.10.2.62:9200/basis_device_info/user/_search?q=channel:B-hicloud

上面第一个根据唯一的_id进行搜索,结果返回0个或者1个文档;第二个通过指定GET方式参数,其中_search和q是ElasticSearch内置的接口关键字,通过指定字段名称和搜索关键词的方式进行搜索,结果以JSON格式返回。

  • Request Body搜索

可以设置请求的body内容,能够支持更加复杂的查询条件然后请求搜索,如下所示:

1 curl -XGET 'http://10.10.2.245:9200/basis_device_info/user/_search' -d '{
2 "query" : {
3 "term" : { "udid": "bc0af2ca66a96725b8b0e0056d4213b6" }
4 }
5 }'

结果示例,如下所示:

1 {"took":11,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":9.45967,"hits":[{"_index":"basis_device_info","_type":"user","_id":"bc0af2ca66a96725b8b0e0056d4213b6","_score":9.45967,"_source":{"installid":"00000FPq","appid":"0","udid":"bc0af2ca66a96725b8b0e0056d4213b6","channel":"B-hicloud","version":"3.1.1","osversion":"4.4.2","device_name":"H60-L02","producer":"HUAWEI","device_type":"0","resolution":"720*1184","screen_size":"4.64","mac":"ec:cb:30:c4:93:e3","idfa":"","idfv":"","imei":"864103021536104","create_time":"2015-01-18 01:29:16"}}]}}
  • 基于Lucene查询语法搜索

如果熟悉Lucene查询(Query),可以构造通过构造复杂的Term关系字符串来进行搜索,示例如下所示:

1 curl -XGET 'http://10.10.2.62:9200/basis_device_info/user/_search' -d '
2 {
3 "query": {
4 "query_string": { "query": "(channel:baidu OR device_name:HUAWEI)" }
5 }
6 }'

查询query字符串的含义是:从channel字段搜索baidu,从device_name字段搜索HUAWEI,然后两者取并集,这实际上一个布尔查询,返回最终结果。

  • 使用multi_match搜索

ElasticSearch支持给定搜索关键词,从多个字段中进行搜索,示例如下所示:

1 curl -XGET 'http://10.10.2.62:9200/basis_device_info/user/_search' -d '
2 {
3 "query": {
4 "multi_match" : {
5 "query": "HTC",
6 "fields": [ "channel", "device_name" ]
7 }
8 }
9 }'

这样,只要在channel和device_name两个字段中出现关键词HTC,则都返回结果,结果应该是两个字段匹配上的文档集合的并集。

  • 支持Filter搜索

可以在制定Filter进行搜索。例如下面是一个按照时间范围进行过滤,得到搜索结果的查询:

01 curl -XGET 'http://10.10.2.62:9200/basis_device_info/user/_search' -d '
02 {
03 "query": {
04 "filtered": {
05 "query": { "match_all": {} },
06 "filter" : {
07 "range" : {
08 "create_time" : { "from" : "2015-01-16 00:00:00", "to" : "2015-01-16 23:59:59" }
09 }
10 }
11 }
12 }
13 }'
  • 分页搜索

ElasticSearch支持分页搜索,可以通过在RESTful连接中指定size和from参数,来进行分页搜索,如下所示:

01 curl -XGET 'http://10.10.2.62:9200/basis_device_info/user/_search?size=10&from=20' -d '
02 {
03 "query": {
04 "filtered": {
05 "query": { "match_all": {} },
06 "filter" : {
07 "range" : {
08 "create_time" : { "from" : "2015-01-16 00:00:00", "to" : "2015-01-16 23:59:59" }
09 }
10 }
11 }
12 }
13 }'

上面搜索的含义是:按照时间范围搜索,从第20个文档开始,返回10个文档,相当于一页取10个文档。

Java客户端

如果熟悉Java语言,而不想使用脚本等其他方式操作ElasticSearch搜索集群,则可以使用ElasticSearch提供的Java客户端API来编码实现,能够更加灵活地控制。ElasticSearch提供的Java客户端支持全部常用操作,如更新索引、索引文档、搜索文档、删除索引等等操作,而且还支持其他一些功能,如同步异步模式、explain查询等,下面我们通过代码来了解一下。
如果使用Maven管理Java代码,可以在pom.xml文件中加入如下依赖:

1 <dependency>
2 <groupId>org.elasticsearch</groupId>
3 <artifactId>elasticsearch</artifactId>
4 <version>2.0.0</version>
5 </dependency>

创建一个ElasticSearch客户端,代码如下所示:

1 // create & configure client
2 Settings settings = Settings.settingsBuilder()
3 .put("cluster.name", "dw_search_engine")
4 .put("client.transport.sniff", true)
5 .build();
6 final Client client = TransportClient.builder().settings(settings).build()
7 .addTransportAddress(newAddress("es-01", 9300))
8 .addTransportAddress(newAddress("es-02", 9300));

可以将你的ElasticSearch集群的节点通过上面的addTransportAddress方法,都与Client对象关联起来,这样在操作ElasticSearch集群中的索引/更新/删除/搜索文档的时候,就能够自动感知。上面newAddress方法如下:

1 private static InetSocketTransportAddress newAddress(String host, int port) throwsUnknownHostException {
2 return new InetSocketTransportAddress(InetAddress.getByName(host), port);
3 }

另外,也可以通过在配置文件elasticsearch.yml中指定相关配置,例如:

1 cluster.name: dw_search_engine
2 client.transport.sniff: true
3 client.transport.ping_timeout: 10s
4 client.transport.nodes_sampler_interval: 10s

那么,创建客户端需要从配置文件中读取配置内容,具体可以查看官方文档。

  • 准备工作

索引的时候,我们是从一个本地文件中读取数据,并构建索引文档需要的格式,然后请求ElasticSearch集群执行索引操作,下面代码是一些基本准备工作:

1 final String index = "basis_device_info";
2 final String type = "user";
3
4 // index documents
5 String f = "C:\\Users\\yanjun\\Desktop\\basis_device_info.txt";
6 File in = new File(f);

从文件中,每次读取一行记录,然后构建一个JSON格式字符串,通过XContentBuilder来表示,代码如下所示:

01 protected static XContentBuilder createSource(String[] a) throws