WhitespaceTokenizer
constructor()
Construct a new WhitespaceTokenizer.
Construct a new WhitespaceTokenizer using a given [ ].
Parameters
factory
the attribute factory to use for this Tokenizer
Construct a new WhitespaceTokenizer using a given max token length
Parameters
maxTokenLen
maximum token length the tokenizer will emit. Must be greater than 0 and less than MAX_TOKEN_LENGTH_LIMIT (1024*1024)
Throws
if maxTokenLen is invalid.
Construct a new WhitespaceTokenizer using a given [ ].
Parameters
factory
the attribute factory to use for this Tokenizer
maxTokenLen
maximum token length the tokenizer will emit. Must be greater than 0 and less than MAX_TOKEN_LENGTH_LIMIT (1024*1024)
Throws
if maxTokenLen is invalid.