Chars in class \p{Line_Break = Complex_Context} are from South East Asian scripts (Thai, Lao, Myanmar, Khmer, etc.). Sequences of these are kept together as as a single token rather than broken up, because the logic required to break them at word boundaries is too complex for UAX#29.

URL_TYPE

const val URL_TYPE: Int

URL token type

WORD_TYPE

const val WORD_TYPE: Int

Alphanumeric sequences

YYEOF

const val YYEOF: Int

This character denotes the end of file.

YYINITIAL

const val YYINITIAL: Int = 0

Lexical States.