NGramTokenizer

Tokenizes the input into n-grams of the given size(s).

On the contrary to [NGramTokenFilter], this class sets offsets so that characters between startOffset and endOffset in the original stream are the same as the term chars.

For example, "abcde" would be tokenized as (minGram=2, maxGram=3):

This tokenizer changed a lot in Lucene 4.4 in order to:

• tokenize in a streaming fashion to support streams which are larger than 1024 chars (limit of the previous version),
• count grams based on unicode code points instead of java chars (and never split in the middle of surrogate pairs),
• give the ability to [isTokenChar] pre-tokenize the stream before computing n-grams.

Additionally, this class doesn't trim trailing whitespaces and emits tokens in a different order, tokens are now emitted by increasing start offsets while they used to be emitted by increasing lengths (which prevented from supporting large input streams).

Inheritors