core/org.gnit.lucenekmp.search.similarities/LMJelinekMercerSimilarity

LMJelinekMercerSimilarity

class LMJelinekMercerSimilarity : LMSimilarity

Language model based on the Jelinek-Mercer smoothing method. From Chengxiang Zhai and John Lafferty. 2001. A study of smoothing methods for language models applied to Ad Hoc information retrieval. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR '01). ACM, New York, NY, USA, 334-342.

The model has a single parameter, λ. According to said paper, the optimal value depends on both the collection and the query. The optimal value is around 0.1 for title queries and 0.7 for long queries.

Values should be between 0 (exclusive) and 1 (inclusive). Values near zero act score more like a conjunction (coordinate level matching), whereas values near 1 behave the opposite (more like pure disjunction).

Constructors

LMJelinekMercerSimilarity

constructor(collectionModel: LMSimilarity.CollectionModel, lambda: Float)

Instantiates with the specified collectionModel and λ parameter.

constructor(collectionModel: LMSimilarity.CollectionModel, discountOverlaps: Boolean, lambda: Float)

Instantiates with the specified collectionModel and parameters.

constructor(lambda: Float)

Instantiates with the specified λ parameter.

Properties

discountOverlaps

val discountOverlaps: Boolean

True if overlap tokens (tokens with a position of increment of zero) are discounted from the document's length.

lambda

val lambda: Float

The λ parameter.

name

open override val name: String

Returns the name of the LM method. The values of the parameters should be included as well.

Functions

computeNorm

open fun computeNorm(state: FieldInvertState): Long

Computes the normalization value for a field at index-time.

scorer

open override fun scorer(boost: Float, collectionStats: CollectionStatistics, vararg termStats: TermStatistics): Similarity.SimScorer

Compute any collection-level weight (e.g. IDF, average document length, etc) needed for scoring a query.

toString

open override fun toString(): String

Returns the name of the LM method. If a custom collection model strategy is used, its name is included as well.