ES搜索排序,文档相关度评分介绍——Field-length norm

简介:

Field-length norm

How long is the field? The shorter the field, the higher the weight. If a term appears in a short field, such as a title field, it is more likely that the content of that field is about the term than if the same term appears in a much bigger body field. The field length norm is calculated as follows:

norm(d) = 1 / √numTerms 

The field-length norm (norm) is the inverse square root of the number of terms in the field.

While the field-length norm is important for full-text search, many other fields don’t need norms. Norms consume approximately 1 byte per string field per document in the index, whether or not a document contains the field. Exact-value not_analyzed string fields have norms disabled by default, but you can use the field mapping to disable norms on analyzed fields as well:

PUT /my_index
{
  "mappings": { "doc": { "properties": { "text": { "type": "string", "norms": { "enabled": false }  } } } } }

This field will not take the field-length norm into account. A long field and a short field will be scored as if they were the same length.

For use cases such as logging, norms are not useful. All you care about is whether a field contains a particular error code or a particular browser identifier. The length of the field does not affect the outcome. Disabling norms can save a significant amount of memory.

Putting it together

These three factors—term frequency, inverse document frequency, and field-length norm—are calculated and stored at index time. Together, they are used to calculate the weight of a single term in a particular document.

Tip

When we refer to documents in the preceding formulae, we are actually talking about a field within a document. Each field has its own inverted index and thus, for TF/IDF purposes, the value of the field is the value of the document.

When we run a simple term query with explain set to true (see Understanding the Score), you will see that the only factors involved in calculating the score are the ones explained in the preceding sections:

PUT /my_index/doc/1 { "text" : "quick brown fox" } GET /my_index/doc/_search?explain { "query": { "term": { "text": "fox" } } }

The (abbreviated) explanation from the preceding request is as follows:

weight(text:fox in 0) [PerFieldSimilarity]:  0.15342641 
result of:
    fieldWeight in 0                         0.15342641
    product of:
        tf(freq=1.0), with freq of 1:        1.0 
        idf(docFreq=1, maxDocs=1):           0.30685282 
        fieldNorm(doc=0):                    0.5 

The final score for term fox in field text in the document with internal Lucene doc ID 0.

The term fox appears once in the text field in this document.

The inverse document frequency of fox in the text field in all documents in this index.

The field-length normalization factor for this field.

Of course, queries usually consist of more than one term, so we need a way of combining the weights of multiple terms. For this, we turn to the vector space model.

 

 










本文转自张昺华-sky博客园博客,原文链接:http://www.cnblogs.com/bonelee/p/6474084.html ,如需转载请自行联系原作者



相关文章
|
2月前
|
自然语言处理 关系型数据库 MySQL
match如何在mysql数据库里进行文本的相似度排序?
【9月更文挑战第1天】match如何在mysql数据库里进行文本的相似度排序?
112 1
|
人工智能 Java
Elasticsearch:使用 function_score 中的weight和gauss衰减函数定制搜索结果的分数
Elasticsearch:使用 function_score 中的weight和gauss衰减函数定制搜索结果的分数
|
SQL Java
白话Elasticsearch04- 结构化搜索之使用terms query搜索多个值以及多值搜索结果优化
白话Elasticsearch04- 结构化搜索之使用terms query搜索多个值以及多值搜索结果优化
526 0
|
SQL 索引
白话Elasticsearch03- 结构化搜索之基于bool组合多个filter条件来搜索数据
白话Elasticsearch03- 结构化搜索之基于bool组合多个filter条件来搜索数据
301 0
|
PHP
sphinx(九)根据关键词相似度排序
全文检索通过sphinx搜索出来的内容是没有问题的。这里使用了PHP内置的similar_text函数对文章的描述以及标题进行相似度的计算,然后根据计算之后的相似度数值进行倒叙排序。
129 0
sphinx(九)根据关键词相似度排序
【1141】PAT Ranking of Institutions (25分)【排序 map】
【1141】PAT Ranking of Institutions (25分)【排序 map】 【1141】PAT Ranking of Institutions (25分)【排序 map】
98 0