AnyTokenizer (PMD Core 6.28.0 API)

java.lang.Object
- net.sourceforge.pmd.cpd.AnyTokenizer

All Implemented Interfaces:

Tokenizer
```
public class AnyTokenizer
extends Object
implements Tokenizer
```
Simple tokenization into words and separators. Can ignore end-of-line comments and recognize double/single quoted string literals. It is not a goal to be very customizable, or have very high quality. Higher-quality lexers should be implemented with a lexer generator.
In PMD 7, this replaces AbstractTokenizer, which provided nearly no more functionality.

- Field Summary
  - Fields inherited from interface net.sourceforge.pmd.cpd.Tokenizer
    DEFAULT_SKIP_BLOCKS_PATTERN, IGNORE_ANNOTATIONS, IGNORE_IDENTIFIERS, IGNORE_LITERALS, IGNORE_USINGS, OPTION_SKIP_BLOCKS, OPTION_SKIP_BLOCKS_PATTERN
- Constructor Summary
  
  Constructors
  Constructor Description
  
  AnyTokenizer()
  
  AnyTokenizer(String eolCommentStart)
- Method Summary
  
  All Methods Instance Methods Concrete Methods
  Modifier and Type Method Description
  
  void tokenize(SourceCode sourceCode, Tokens tokenEntries)
  - Methods inherited from class java.lang.Object
    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AnyTokenizer
```
public AnyTokenizer()
```

AnyTokenizer

public AnyTokenizer(String eolCommentStart)

Method Detail

tokenize

public void tokenize(SourceCode sourceCode,
                     Tokens tokenEntries)

Specified by:: tokenize in interface Tokenizer