common/org.gnit.lucenekmp.analysis.shingle/FixedShingleFilter

FixedShingleFilter

class FixedShingleFilter : GraphTokenFilter

A FixedShingleFilter constructs shingles (token n-grams) from a token stream. In other words, it creates combinations of tokens as a single token.

Unlike the [ShingleFilter], FixedShingleFilter only emits shingles of a fixed size, and never emits unigrams, even at the end of a TokenStream. In addition, if the filter encounters stacked tokens (eg synonyms), then it will output stacked shingles

For example, the sentence "please divide this sentence into shingles" might be tokenized into shingles "please divide", "divide this", "this sentence", "sentence into", and "into shingles".

This filter handles position increments > 1 by inserting filler tokens (tokens with termtext "_").

Constructors

FixedShingleFilter

constructor(input: TokenStream, shingleSize: Int)

Creates a FixedShingleFilter over an input token stream

constructor(input: TokenStream, shingleSize: Int, tokenSeparator: String, fillerToken: String)

Creates a FixedShingleFilter over an input token stream

Types

object Companion

Properties

attributeClassesIterator

val attributeClassesIterator: Iterator<Any>

attributeFactory

val attributeFactory: AttributeFactory

attributeImplsIterator

val attributeImplsIterator: Iterator<AttributeImpl>

Functions

fun <T : Attribute> addAttribute(attClass: KClass<T>): T

addAttributeImpl

fun addAttributeImpl(att: AttributeImpl)

cachedTokenCount

fun cachedTokenCount(): Int

fun captureState(): AttributeSource.State?

clearAttributes

fun clearAttributes()

cloneAttributes

fun cloneAttributes(): AttributeSource

open override fun close()

fun copyTo(target: AttributeSource)

open override fun end()

fun endAttributes()

open operator override fun equals(obj: Any?): Boolean

fun <T : Attribute> getAttribute(attClass: KClass<T>): T?

getTrailingPositions

fun getTrailingPositions(): Int

fun hasAttribute(attClass: KClass<out Attribute>): Boolean

fun hasAttributes(): Boolean

open override fun hashCode(): Int

incrementBaseToken

fun incrementBaseToken(): Boolean

fun incrementGraph(): Boolean

incrementGraphToken

fun incrementGraphToken(): Boolean

open override fun incrementToken(): Boolean

reflectAsString

fun reflectAsString(prependAttClass: Boolean): String

fun reflectWith(reflector: AttributeReflector)

removeAllAttributes

fun removeAllAttributes()

open override fun reset()

fun restoreState(state: AttributeSource.State?)

open override fun toString(): String

open override fun unwrap(): TokenStream