StandardTokenizer

Creates a new instance of the org.gnit.lucenekmp.analysis.standard.StandardTokenizer. Attaches the input to the newly created JFlex scanner!!.

Creates a new StandardTokenizer with a given org.gnit.lucenekmp.util.AttributeFactory

The caller must pass in a Class value. This method first checks if an instance of that class is already in this AttributeSource and returns it. Otherwise a new instance is created, added to this AttributeSource and returned.

Expert: Adds a custom AttributeImpl instance with one or more Attribute interfaces.

Captures the state of all Attributes. The return value can be passed to .restoreState to restore the state of this or another AttributeSource.

Resets all Attributes in this AttributeSource by calling AttributeImpl.clear on each Attribute implementation.

Performs a clone of all AttributeImpl instances returned in a new AttributeSource instance. This method can be used to e.g. create another TokenStream with exactly the same attributes (using .AttributeSource). You can also use it as a (non-performant) replacement for .captureState, if you need to look into / modify the captured state.

{@inheritDoc}

Copies the contents of this AttributeSource to the given target AttributeSource. The given instance has to provide all Attributes this instance contains. The actual attribute implementations must be identical in both AttributeSource instances; ideally both AttributeSource instances should use the same [ ]. You can use this method as a replacement for .restoreState, if you use .cloneAttributes instead of .captureState.

This method is called by the consumer after the last token has been consumed, after .incrementToken returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.

Resets all Attributes in this AttributeSource by calling AttributeImpl.end on each Attribute implementation.

Returns the instance of the passed in Attribute contained in this AttributeSource

Returns the current maximum token length

The caller must pass in a Class value. Returns true, iff this AttributeSource contains the passed-in Attribute.

Returns true, iff this AttributeSource has any attributes

Consumers (i.e., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate [ ]s with the attributes of the next token.

This method returns the current attribute values as a string in the following format by calling the .reflectWith method:

This method is for introspection of attributes, it should simply add the key/values this AttributeSource holds to the given AttributeReflector.

Removes all attributes and their implementations from this AttributeSource.

This method is called by a consumer before it begins consumption using .incrementToken.

Restores this state by copying the values of all attribute implementations that this state contains into the attributes implementations of the targetStream. The targetStream must contain a corresponding instance for each argument contained in this state (e.g. it is not possible to restore the state of an AttributeSource containing a TermAttribute into a AttributeSource using a Token instance as implementation).

Set the max allowed token length. Tokens larger than this will be chopped up at this token length and emitted as multiple tokens. If you need to skip such large tokens, you could increase this max length, and then use LengthFilter to remove long tokens. The default is StandardAnalyzer.DEFAULT_MAX_TOKEN_LENGTH.

Expert: Set a new reader on the Tokenizer. Typically, an analyzer (in its tokenStream method) will use this to re-use a previously created tokenizer.

Returns a string consisting of the class's simple name, the hex representation of the identity hash code, and the current reflection of all attributes.