AnyTokenizer (PMD Core 7.0.0-rc4 API)

java.lang.Object
- net.sourceforge.pmd.cpd.AnyTokenizer

All Implemented Interfaces:

Tokenizer
```
public class AnyTokenizer
extends Object
implements Tokenizer
```
Simple tokenization into words and separators. Can ignore end-of-line comments and recognize double/single quoted string literals. It is not a goal to be very customizable, or have very high quality. Higher-quality lexers should be implemented with a lexer generator.
In PMD 7, this replaces AbstractTokenizer, which provided nearly no more functionality.

Constructor Summary

Constructors
Constructor Description

AnyTokenizer()

AnyTokenizer(String eolCommentStart)

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`void`	`tokenize(TextDocument document, TokenFactory tokens)`	Tokenize the source code and record tokens using the provided token factory.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - AnyTokenizer
```
public AnyTokenizer()
```
  - AnyTokenizer
```
public AnyTokenizer(String eolCommentStart)
```
- Method Detail
  - tokenize
```
public void tokenize(TextDocument document,
                     TokenFactory tokens)
```
    Description copied from interface: Tokenizer
    
    Tokenize the source code and record tokens using the provided token factory.
    
    Specified by:
    
    tokenize in interface Tokenizer