Using the UnderC Tokenizer Class



It s often necessary to parse complex text files, where standard i/o

is too clumsy. C programmers fall back on strtok(), but this can be

tricky to use properly. Besides, you are still responsible for keeping

strtok() fed with new input, and I don t like the schlepp.



Tokenizer is a text-parsing input stream, modelled after the (undocumented)

VCL TParser class, and based on the UnderC tokenizing preprocessor front-end.