core/org.gnit.lucenekmp.analysis.standard/StandardTokenizer/StandardTokenizer

StandardTokenizer

constructor()

Creates a new instance of the org.gnit.lucenekmp.analysis.standard.StandardTokenizer. Attaches the input to the newly created JFlex scanner!!.

See http://issues.apache.org/jira/browse/LUCENE-1068

constructor(factory: AttributeFactory)

Creates a new StandardTokenizer with a given org.gnit.lucenekmp.util.AttributeFactory