StreamTokenizer

Creates a stream tokenizer that parses the specified input stream. The stream tokenizer is initialized to the following default state:

Create a tokenizer that parses the given character stream.

If the current token is a number, this field contains the value of that number. The current token is a number when the value of the ttype field is TT_NUMBER.

If the current token is a word token, this field contains a string giving the characters of the word token. When the current token is a quoted string token, this field contains the body of the string.

After a call to the nextToken method, this field contains the type of the token just read. For a single character token, its value is the single character, converted to an integer. For a quoted string token, its value is the quote character. Otherwise, its value is one of the following:

Specifies that the character argument starts a single-line comment. All characters from the comment character to the end of the line are ignored by this stream tokenizer.

Determines whether or not ends of line are treated as tokens. If the flag argument is true, this tokenizer treats end of lines as tokens; the nextToken method returns TT_EOL and also sets the ttype field to this value when an end of line is read.

Returns the current line number.

Determines whether or not word token are automatically lowercased. If the flag argument is true, then the value in the sval field is lowercased whenever a word token is returned (the ttype field has the value TT_WORD) by the nextToken method of this tokenizer.

Parses the next token from the input stream of this tokenizer. The type of the next token is returned in the ttype field. Additional information about the token may be in the nval field or the sval field of this tokenizer.

Specifies that the character argument is "ordinary" in this tokenizer. It removes any special significance the character has as a comment character, word component, string delimiter, white space, or number character. When such a character is encountered by the parser, the parser treats it as a single-character token and sets ttype field to the character value.

Specifies that all characters c in the range low <= c <= high are "ordinary" in this tokenizer. See the ordinaryChar method for more information on a character being ordinary.

Specifies that numbers should be parsed by this tokenizer. The syntax table of this tokenizer is modified so that each of the twelve characters:

Causes the next call to the nextToken method of this tokenizer to return the current value in the ttype field, and not to modify the value in the nval or sval field.

Specifies that matching pairs of this character delimit string constants in this tokenizer.

Resets this tokenizer's syntax table so that all characters are "ordinary." See the ordinaryChar method for more information on a character being ordinary.

Determines whether or not the tokenizer recognizes C++-style comments. If the flag argument is true, this stream tokenizer recognizes C++-style comments. Any occurrence of two consecutive slash characters ('/') is treated as the beginning of a comment that extends to the end of the line.

Determines whether or not the tokenizer recognizes C-style comments. If the flag argument is true, this stream tokenizer recognizes C-style comments. All text between successive occurrences of / * and */ are discarded.

Returns the string representation of the current stream token and the line number it occurs on.

Specifies that all characters c in the range low <= c <= high are white space characters. White space characters serve only to separate tokens in the input stream.

Specifies that all characters c in the range low <= c <= high are word constituents. A word token consists of a word constituent followed by zero or more word constituents or number constituents.