- All Implemented Interfaces:
Simple tokenization into words and separators. Can ignore end-of-line comments and recognize double/single quoted string literals. It is not a goal to be very customizable, or have very high quality. Higher-quality lexers should be implemented with a lexer generator.
In PMD 7, this replaces AbstractTokenizer, which provided nearly no more functionality.