New Lucene 4 Functions Improve Enterprise Search Indexing
原文链接http://java.dzone.com/articles/new-index-statistics-lucene-40
In the past, Lucene recorded only the bare minimal aggregate index statistics necessary to support its hard-wiredclassic vector space scoring model.
Fortunately, this situation iswildly improved in trunk(to be 4.0), where wehave a selection of modern scoring models, includingOkapi BM25,Language models,Divergence from Randomness modelsandInformation-based models. To support these, we now save a number of commonly used index statistics per index segment, and make them available at search time.
To understand the new statistics, let's pretend we've indexed the following two example documents, each with only one field "title":
· document 1: The Lion, the Witch, and the Wardrobe
· document 2: The Da Vinci Code
Assume we tokenize on whitespace, commas are removed, all terms are downcased and we don't discard stop-words. Here are the statistics Lucene tracks:
TermsEnum.docFreq()
How many documents contain at least one occurrence of the term in the field; 3.x indices also save this (TermEnum.docFreq()). For term "lion" docFreq is 1, and for term "the" it's 2.
Terms.getSumDocFreq()
Number of postings, i.e. sum of TermsEnum.docFreq() across all terms in the field. For our example documents this is 9.
TermsEnum.totalTermFreq()
Number of occurrences of this term in the field, across all documents. For term "the" it's 4, for term "vinci" it's 1.
Terms.getSumTotalTermFreq()
Number of term occurrences in the field, across all documents; this is the sum of TermsEnum.totalTermFreq() across all unique terms in the field. For our example documents this is 11.
Terms.getDocCount()
How many documents have at least one term for this field. In our example documents, this is 2, but if for example one of the documents was missing the title field, it would be 1.
Terms.getUniqueTermCount()
How many unique terms were seen in this field. For our example documents this is 8. Note that this statistic is of limited utility for scoring, because it's only available per-segment and you cannot (efficiently!) compute this across all segments in the index (unless there is only one segment).
Fields.getUniqueTermCount()
Number of unique terms across all fields; this is the sum of Terms.getUniqueTermCount() across all fields. In our example documents this is 8. Note that this is also only available per-segment.
Fields.getUniqueFieldCount()
Number of unique fields. For our example documents this is 1; if we also had a body field and an abstract field, it would be 3. Note that this is also only available per-segment.
3.x indices only store TermsEnum.docFreq(), so if you want to experiment with the new scoring models in Lucene 4.0, you should either re-index or upgrade your index using IndexUpgrader. Note that the new scoring models all use the same single-byte norms format, so you can freely switch between them without re-indexing.
In addition to what's stored in the index, there are also these statistics available per-field, per-document while indexing, in the FieldInvertState passed to Similarity.computeNorm method for both 3.x and 4.0:
length
How many tokens in the document. For document 1 it's 7; for document 2 it's 4.
uniqueTermCount
For this field in this document, how many unique terms are there? For document 1, it's 5; for document 2 it's 4.
maxTermFrequency
What was the count for the most frequent term in this document. For document 1 it's 3 ("the" occurs 3 times); for document 2 it's 1.
In 3.x, if you want to consume these indexing-time statistics, you'll have to save them away yourself (e.g., somehow encoding them into the single-byte norm value). However, since 4.0 usesdoc valuesfor norms, you have more freedom to encode these statistics however you'd like. Your custom similarity can then pull from these.
From these available statistics you're now free to derive other commonly used statistics:
· Average document length is Terms.getSumTotalTermFreq() divided by Terms.getDocCount().
· Average within-document term frequency is FieldInvertState.length divided by FieldInvertState.uniqueTermCount.
· Average document length across the collection is Terms.getSumTotalTermFreq() divided by maxDoc (or Terms.getDocCount(), if not all documents have the field).
· Average number of unique terms per document is Terms.getSumDocFreq() divided by maxDoc (or Terms.getDocCount(field), if not all documents have the field).
Remember that the statistics do not reflect deleted documents, until those documents are merged away; in general this also means that segment merging will alter scores! Similarly, if the field omits term frequencies, then the statistics will not be correct (though they will still be consistent with one another: we will pretend each term occurred once per document).
Published at DZone with permission ofMichael Mccandless, author and DZone MVB. (source)
(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, In
