关于TrieField的全面认识、理解、运用

本文涉及的产品
服务治理 MSE Sentinel/OpenSergo,Agent数量 不受限
任务调度 XXL-JOB 版免费试用,400 元额度,开发版规格
注册配置 MSE Nacos/ZooKeeper,182元/月
简介: 关于trieField的理解补充下3篇文档,相当的系统、全面!看相关文档连接,不解释。 http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/api/all/org/apache/lucene/search/NumericR

关于trieField的理解补充下3篇文档,相当的系统、全面!看相关文档连接,不解释。


http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/api/all/org/apache/lucene/search/NumericRangeQuery.html

http://blog.csdn.net/fancyerii/article/details/7256379

http://hadoopcn.iteye.com/blog/1550402

http://rdc.taobao.com/team/jm/archives/1699

 


 
extends 
MultiTermQuery

A 
Query
 that matches numeric values within a
specified range. To use this, you must first index the numeric
values using 
NumericField
 (expert: 
NumericTokenStream
). If your terms are instead
textual, you should use 
TermRangeQuery
. 
NumericRangeFilter
 is the filter equivalent of
this query.

You create a new NumericRangeQuery with the static factory
methods, eg:

 

matches all documents whose float valued "weight" field ranges
from 0.03 to 0.10, inclusive.

The performance of NumericRangeQuery is much better than the
corresponding 
TermRangeQuery
 because the number of terms that
must be searched is usually far fewer, thanks to trie indexing,
described below.

You can optionally specify a 
precisionStep
 when creating this query. This is
necessary if you've changed this configuration from its default (4)
during indexing. Lower values consume more disk space but speed up
searching. Suitable values are between 1 and 8. A
good starting point to test is 4, which is the default value
for all Numeric* classes. See 
below
 for details.

This query defaults to 
MultiTermQuery.CONSTANT_SCORE_AUTO_REWRITE_DEFAULT
 for 32 bit
(int/float) ranges with precisionStep ≤8 and 64 bit (long/double)
ranges with precisionStep ≤6. Otherwise it uses 
MultiTermQuery.CONSTANT_SCORE_FILTER_REWRITE
 as the number of
terms is likely to be high. With precision steps of ≤4, this query
can be run with one of the BooleanQuery rewrite methods without
changing BooleanQuery's default max clause count.

How it works

See the publication about panFMP, where this algorithm was described
(referred to as TrieRangeQuery):

Schindler, U, Diepenbroek, M, 2008.
Generic XML-based Framework for Metadata Portals.
Computers & Geosciences 34 (12), 1947-1955. doi:10.1016/j.cageo.2008.02.023

A quote from this paper: Because Apache Lucene is a
full-text search engine and not a conventional database, it cannot
handle numerical ranges (e.g., field value is inside user defined
bounds, even dates are numerical values). We have developed an
extension to Apache Lucene that stores the numerical values in a
special string-encoded format with variable precision (all
numerical values like doubles, longs, floats, and ints are
converted to lexicographic sortable string representations and
stored with different precisions (for a more detailed description
of how the values are stored, see 
NumericUtils
). A range is then divided recursively
into multiple intervals for searching: The center of the range is
searched only with the lowest possible precision in the
trie, while the boundaries are matched more exactly. This
reduces the number of terms dramatically.

For the variant that stores long values in 8 different
precisions (each reduced by 8 bits) that uses a lowest precision of
1 byte, the index contains only a maximum of 256 distinct values in
the lowest precision. Overall, a range could consist of a
theoretical maximum of 7*255*2 + 255 = 3825 distinct
terms (when there is a term for every distinct value of an
8-byte-number in the index and the range covers almost all of them;
a maximum of 255 distinct values is used because it would always be
possible to reduce the full 256 values to one term with degraded
precision). In practice, we have seen up to 300 terms in most cases
(index with 500,000 metadata records and a uniform value
distribution).

Precision
Step

You can choose any precisionStep when encoding
values. Lower step values mean more precisions and so more terms in
index (and index gets larger). On the other hand, the maximum
number of terms to match reduces, which optimized query speed. The
formula to calculate the maximum term count is:

 

(this formula is only correct, when
bitsPerValue/precisionStep is an integer; in other
cases, the value must be rounded up and the last summand must
contain the modulo of the division as precision step)
. For
longs stored using a precision step of 4, n = 15*15*2 + 15 =
465
, and for a precision step of 2, n = 31*3*2 + 3 =
189
. But the faster search speed is reduced by more seeking
in the term enum of the index. Because of this, the ideal
precisionStep value can only be found out by testing.
Important: You can index with a lower precision step value
and test search speed using a multiple of the original step
value.

Good values for precisionStep are depending on
usage and data type:

  • The default for all data types is 4, which is used, when
    no precisionStep is given.
  • Ideal value in most cases for 64 bit data types
    (long, double) is 6 or 8.
  • Ideal value in most cases for 32 bit data types
    (int, float) is 4.
  • For low cardinality fields larger precision steps are good. If
    the cardinality is < 100, it is fair to use 
    Integer.MAX_VALUE
     (see below).
  • Steps ≥64 for long/double and ≥32 for
    int/float produces one token per value in the index and
    querying is as slow as a conventional 
    TermRangeQuery
    . But it can be used to produce
    fields, that are solely used for sorting (in this case simply use

    Integer.MAX_VALUE
     as precisionStep).
    Using 
    NumericFields
     for sorting is ideal, because
    building the field cache is much faster than with text-only
    numbers. These fields have one term per value and therefore also
    work with term enumeration for building distinct lists (e.g. facets
    / preselected values to search for). Sorting is also possible with
    range query optimized fields using one of the above
    precisionSteps.

Comparisons of the different types of RangeQueries on an index
with about 500,000 docs showed that 
TermRangeQuery
 in boolean rewrite mode (with
raised 
BooleanQuery
 clause count) took about 30-40 secs
to complete, 
TermRangeQuery
 in constant score filter rewrite
mode took 5 secs and executing this class took <100ms to
complete (on an Opteron64 machine, Java 1.5, 8 bit precision step).
This query type was developed for a geographic portal, where the
performance for e.g. bounding boxes or exact date/time stamps is
important.

Since:
2.9
See Also:

Serialized Form
相关文章
|
并行计算 算法 C++
统一内存统一内存的基本概念和使用
统一内存统一内存的基本概念和使用
2155 0
统一内存统一内存的基本概念和使用
|
11月前
|
机器学习/深度学习 人工智能 算法
AI技术在医疗领域的应用及挑战
【10月更文挑战第4天】本文将探讨AI技术在医疗领域的应用及其面临的挑战。我们将从AI技术的定义和发展历程入手,分析其在医疗领域的应用场景,包括辅助诊断、个性化治疗、药物研发等方面。同时,我们也将讨论AI技术在医疗领域面临的挑战,如数据隐私、算法偏见等问题。最后,我们将以一个简单的代码示例来展示AI技术在医疗领域的应用。
255 0
|
传感器 存储 安全
机器通信 | 《5G移动无线通信技术》之八
本节主要介绍了机器通信的内容以及超可靠机器类通信。
机器通信  | 《5G移动无线通信技术》之八
|
算法
FM算法介绍
概述 FM (Factorization Machine) 算法可进行回归和二分类预测,它的特点是考虑了特征之间的相互作用,是一种非线性模型,目前FM算法是推荐领域被验证的效果较好的推荐方案之一,在诸多电商、广告、直播厂商的推荐领域有广泛应用。
12065 0
|
Web App开发 移动开发 JavaScript
VS Code 自定义快捷输入
位置 ctrl+shift+p 搜索: snippets 输入类型: 比如 html、javascript 参数说明 prefix:使用代码段的快捷入口 body:需要设置的代码放在这里,字符串间换行的话使用\r\n换行符隔开.
2217 0
|
Web App开发 开发者 iOS开发
新版本浏览器为何无法访问部分本地项目
情况 Chrome 以及Safari 浏览器在最近的自动升级之后,在地址栏输入本地项目地址(simple.app)时,显示: 新版本浏览器无法访问部分本地项目,对开发者有相当大的影响 新版本浏览器无法访问部分本地项目,对开发者有相当大的影响 Chrome 版本为63.0,Safari 的版本为11.0.2。
1176 0
|
供应链 算法 安全
阿里云E2 云采销:云采销产品整体介绍|学习笔记(一)
快速学习阿里云E2 云采销:云采销产品整体介绍。
阿里云E2 云采销:云采销产品整体介绍|学习笔记(一)
|
存储 芯片
#计算机组成原理# chapter 4-存储器-4.2主存储器 4.2.1主存储器概述&&4.2.2半导体存储芯片简介
#计算机组成原理# chapter 4-存储器-4.2主存储器 4.2.1主存储器概述&&4.2.2半导体存储芯片简介
828 0
#计算机组成原理# chapter 4-存储器-4.2主存储器 4.2.1主存储器概述&&4.2.2半导体存储芯片简介
|
Java 测试技术 Maven
Java 反编译工具的使用与对比分析(一)
Java 反编译工具的使用与对比分析
857 0
Java 反编译工具的使用与对比分析(一)
官宣!支付宝小程序的 23 个入口大盘点
近日,支付宝小程序场景值文档发布。场景值用于描述用户进入小程序的路径,也就是说,场景值即代表了的小程序入口 。
3796 12
官宣!支付宝小程序的 23 个入口大盘点