CharTokenizer

constructor()

Creates a new CharTokenizer instance

constructor(factory: AttributeFactory)

Creates a new CharTokenizer instance

factory

the attribute factory to use for this Tokenizer

constructor(factory: AttributeFactory, maxTokenLen: Int)

Creates a new CharTokenizer instance

factory

the attribute factory to use for this Tokenizer

maxTokenLen

maximum token length the tokenizer will emit. Must be greater than 0 and less than MAX_TOKEN_LENGTH_LIMIT (1024*1024)

if maxTokenLen is invalid.