.A Token is an occurence of a term from the text of a field. It consists of a term s text, the start and end offset of the term in the text of the field, and a type string. The start and end offsets permit applications to re-associate a token with it

This function is not a part of the C++ standard but is syntactic sugar for internal library use only.

通过flex和C语言程序的配合可以完成语法分析器(The syntactic parser can be accomplished through the combination of flex and C language programs)