common/org.gnit.lucenekmp.analysis.shingle/ShingleFilter

ShingleFilter

A ShingleFilter constructs shingles (token n-grams) from a token stream. In other words, it creates combinations of tokens as a single token.

For example, the sentence "please divide this sentence into shingles" might be tokenized into shingles "please divide", "divide this", "this sentence", "sentence into", and "into shingles".

This filter handles position increments > 1 by inserting filler tokens (tokens with termtext "_"). It does not handle a position increment of 0.

Constructors

ShingleFilter

constructor(input: TokenStream, minShingleSize: Int, maxShingleSize: Int)

Constructs a ShingleFilter with the specified shingle size from the TokenStream input

constructor(input: TokenStream, maxShingleSize: Int)

Constructs a ShingleFilter with the specified shingle size from the TokenStream input

constructor(input: TokenStream)

Construct a ShingleFilter with default shingle size: 2.

constructor(input: TokenStream, tokenType: String)

Construct a ShingleFilter with the specified token type for shingle tokens and the default shingle size: 2

Types

Companion

object Companion

Properties

attributeClassesIterator

val attributeClassesIterator: Iterator<Any>

attributeFactory

val attributeFactory: AttributeFactory

attributeImplsIterator

val attributeImplsIterator: Iterator<AttributeImpl>

noShingleOutput

var noShingleOutput: Boolean

true if no shingles have been output yet (for outputUnigramsIfNoShingles).

Functions

addAttribute

fun <T : Attribute> addAttribute(attClass: KClass<T>): T

addAttributeImpl

fun addAttributeImpl(att: AttributeImpl)

captureState

fun captureState(): AttributeSource.State?

clearAttributes

fun clearAttributes()

cloneAttributes

fun cloneAttributes(): AttributeSource

open override fun close()

copyTo

fun copyTo(target: AttributeSource)

end

open override fun end()

endAttributes

fun endAttributes()

equals

open operator override fun equals(obj: Any?): Boolean

getAttribute

fun <T : Attribute> getAttribute(attClass: KClass<T>): T?

hasAttribute

fun hasAttribute(attClass: KClass<out Attribute>): Boolean

hasAttributes

fun hasAttributes(): Boolean

hashCode

open override fun hashCode(): Int

incrementToken

open override fun incrementToken(): Boolean

reflectAsString

fun reflectAsString(prependAttClass: Boolean): String

reflectWith

fun reflectWith(reflector: AttributeReflector)

removeAllAttributes

fun removeAllAttributes()

reset

open override fun reset()

restoreState

fun restoreState(state: AttributeSource.State?)

setFillerToken

fun setFillerToken(fillerToken: String?)

Sets the string to insert for each position at which there is no token (i.e., when position increment is greater than one).

setMaxShingleSize

fun setMaxShingleSize(maxShingleSize: Int)

Set the max shingle size (default: 2)

setMinShingleSize

fun setMinShingleSize(minShingleSize: Int)

Set the min shingle size (default: 2).

setOutputUnigrams

fun setOutputUnigrams(outputUnigrams: Boolean)

Shall the output stream contain the input tokens (unigrams) as well as shingles? (default: true.)

setOutputUnigramsIfNoShingles

fun setOutputUnigramsIfNoShingles(outputUnigramsIfNoShingles: Boolean)

Shall we override the behavior of outputUnigrams==false for those times when no shingles are available (because there are fewer than minShingleSize tokens in the input stream)? (default: false.)

setTokenSeparator

fun setTokenSeparator(tokenSeparator: String?)

Sets the string to use when joining adjacent tokens to form a shingle

setTokenType

fun setTokenType(tokenType: String)

Set the type of the shingle tokens produced by this filter. (default: "shingle")

toString

open override fun toString(): String

unwrap

open override fun unwrap(): TokenStream