Package-level declarations

Filters StandardTokenizer with LowerCaseFilter and StopFilter, using a configurable list of stop words.

A grammar-based tokenizer constructed with JFlex.

Factory for StandardTokenizer.

This class implements Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.