incrementToken

Output bigrams whenever possible to optimize queries. Only output unigrams when they are not a member of a bigram. Example:

• input: "the rain in spain falls mainly"
• output:"the-rain", "rain-in" ,"in-spain", "falls", "mainly"