Package net.sourceforge.pmd.cpd
Class AnyTokenizer
- java.lang.Object
-
- net.sourceforge.pmd.cpd.AnyTokenizer
-
- All Implemented Interfaces:
Tokenizer
public class AnyTokenizer extends Object implements Tokenizer
Simple tokenization into words and separators. Can ignore end-of-line comments and recognize double/single quoted string literals. It is not a goal to be very customizable, or have very high quality. Higher-quality lexers should be implemented with a lexer generator.In PMD 7, this replaces AbstractTokenizer, which provided nearly no more functionality.
-
-
Field Summary
-
Fields inherited from interface net.sourceforge.pmd.cpd.Tokenizer
DEFAULT_SKIP_BLOCKS_PATTERN, IGNORE_ANNOTATIONS, IGNORE_IDENTIFIERS, IGNORE_LITERALS, IGNORE_USINGS, OPTION_IGNORE_LITERAL_SEQUENCES, OPTION_SKIP_BLOCKS, OPTION_SKIP_BLOCKS_PATTERN
-
-
Constructor Summary
Constructors Constructor Description AnyTokenizer()
AnyTokenizer(String eolCommentStart)
-
-
-
Constructor Detail
-
AnyTokenizer
public AnyTokenizer()
-
AnyTokenizer
public AnyTokenizer(String eolCommentStart)
-
-
Method Detail
-
tokenize
public void tokenize(SourceCode sourceCode, Tokens tokenEntries)
-
-