SegTokenFilter
class SegTokenFilter
Filters a SegToken by converting full-width latin to half-width, then lowercasing latin. Additionally, all punctuation is converted into Utility.COMMON_DELIMITER
Filters a SegToken by converting full-width latin to half-width, then lowercasing latin. Additionally, all punctuation is converted into Utility.COMMON_DELIMITER