Set the max allowed token length. Tokens larger than this will be chopped up at this token length and emitted as multiple tokens. If you need to skip such large tokens, you could increase this max length, and then use LengthFilter to remove long tokens. The default is StandardAnalyzer.DEFAULT_MAX_TOKEN_LENGTH.

reuseStrategy

val reuseStrategy: Analyzer.ReuseStrategy

Returns the used ReuseStrategy.

stopwords

val stopwords: CharArraySet

An immutable stopword set

storedValue

var storedValue: CloseableThreadLocal<Any>?

Functions

open override fun close()

Frees persistent resources used by this Analyzer

getOffsetGap

open fun getOffsetGap(fieldName: String?): Int

Just like .getPositionIncrementGap, except for Token offsets instead. By default this returns 1. This method is only called if the field produced at least one token for indexing.

getPositionIncrementGap

open fun getPositionIncrementGap(fieldName: String?): Int

Invoked before indexing a IndexableField instance if terms have already been added to that field. This allows custom analyzers to place an automatic position increment gap between IndexbleField instances using the same field name. The default value position increment gap is

normalize

fun normalize(fieldName: String, text: String): BytesRef

Normalize a string down to the representation that it would have in the index.

tokenStream

fun tokenStream(fieldName: String, text: String): TokenStream

Returns a TokenStream suitable for fieldName, tokenizing the contents of text.

fun tokenStream(fieldName: String, reader: Reader): TokenStream

Returns a TokenStream suitable for fieldName, tokenizing the contents of reader.