当你写词法分析器时，Sun s Java中的 StreamTokenizer 类是很有用的。所以我生成了类似的一个类CStringTokenizer ，它的使用方法类似于Java的StreamTokenizer ，也提供了一些额外的功能，函数名也稍微有些不同-law analyzer, Sun's Java s StreamTokenizer category is very useful. I generated a similar type CStringTokenizer, its

This a prototype for the Compiler design basic steps written in java. A tokenizer to get the tokens from the Input code.-This is a prototype for the Compiler design basic steps written in java. A tokenizer to get the tokens from the Input code.