AbstractTokenizer (PMD Core 6.24.0 API)

java.lang.Object
- net.sourceforge.pmd.cpd.AbstractTokenizer

All Implemented Interfaces:

Tokenizer
```
public abstract class AbstractTokenizer
extends Object
implements Tokenizer
```
Author:

Zev Blut zb@ubit.com, Romain PELISSE belaran@gmail.com

Field Summary

Fields
Modifier and Type	Field	Description
`protected List<String>`	`ignorableCharacter`
`protected List<String>`	`ignorableStmt`
`protected char`	`oneLineCommentChar`
`protected Character`	`spanMultipleLinesLineContinuationCharacter`
`protected boolean`	`spanMultipleLinesString`
`protected List<String>`	`stringToken`

Fields inherited from interface net.sourceforge.pmd.cpd.Tokenizer
DEFAULT_SKIP_BLOCKS_PATTERN, IGNORE_ANNOTATIONS, IGNORE_IDENTIFIERS, IGNORE_LITERALS, IGNORE_USINGS, OPTION_SKIP_BLOCKS, OPTION_SKIP_BLOCKS_PATTERN

Constructor Summary

Constructors
Constructor Description

AbstractTokenizer()

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type Method Description

void tokenize(SourceCode tokens, Tokens tokenEntries)
- Methods inherited from class java.lang.Object
  clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

stringToken
```
protected List<String> stringToken
```

ignorableCharacter

protected List<String> ignorableCharacter

ignorableStmt
```
protected List<String> ignorableStmt
```

oneLineCommentChar
```
protected char oneLineCommentChar
```

spanMultipleLinesString

protected boolean spanMultipleLinesString

spanMultipleLinesLineContinuationCharacter

protected Character spanMultipleLinesLineContinuationCharacter

Constructor Detail
- AbstractTokenizer
```
public AbstractTokenizer()
```

Method Detail

tokenize

public void tokenize(SourceCode tokens,
                     Tokens tokenEntries)

Specified by:: tokenize in interface Tokenizer