KoreanTokenizer
constructor()
Creates a new KoreanTokenizer with default parameters.
constructor(factory: AttributeFactory, userDictionary: UserDictionary?, mode: KoreanTokenizer.DecompoundMode, outputUnknownUnigrams: Boolean)
constructor(factory: AttributeFactory, userDictionary: UserDictionary?, mode: KoreanTokenizer.DecompoundMode, outputUnknownUnigrams: Boolean, discardPunctuation: Boolean)
Create a new KoreanTokenizer using the system and unknown dictionaries shipped with Lucene.
constructor(factory: AttributeFactory, systemDictionary: TokenInfoDictionary, unkDictionary: UnknownDictionary, connectionCosts: ConnectionCosts, userDictionary: UserDictionary?, mode: KoreanTokenizer.DecompoundMode, outputUnknownUnigrams: Boolean, discardPunctuation: Boolean)
Create a new KoreanTokenizer supplying a custom system dictionary and unknown dictionary.