All Known Implementing Classes:

AbstractTokenizer, AntlrTokenizer, AnyTokenizer
```
public interface Tokenizer
```

Field Summary

Fields
Modifier and Type	Field	Description
`static String`	`DEFAULT_SKIP_BLOCKS_PATTERN`
`static String`	`IGNORE_ANNOTATIONS`
`static String`	`IGNORE_IDENTIFIERS`
`static String`	`IGNORE_LITERALS`
`static String`	`IGNORE_USINGS`	Ignore using directives in C#.
`static String`	`OPTION_IGNORE_LITERAL_SEQUENCES`	Ignore sequences of literals (e.g, `0,0,0,0...`).
`static String`	`OPTION_SKIP_BLOCKS`	Enables or disabled skipping of blocks like a pre-processor.
`static String`	`OPTION_SKIP_BLOCKS_PATTERN`	Configures the pattern, to find the blocks to skip.

Method Summary

All Methods Instance Methods Abstract Methods
Modifier and Type Method Description

void tokenize(SourceCode sourceCode, Tokens tokenEntries)

- Field Detail
  - IGNORE_LITERALS
```
static final String IGNORE_LITERALS
```
    See Also:
    
    Constant Field Values
  - IGNORE_IDENTIFIERS
```
static final String IGNORE_IDENTIFIERS
```
    See Also:
    
    Constant Field Values
  - IGNORE_ANNOTATIONS
```
static final String IGNORE_ANNOTATIONS
```
    See Also:
    
    Constant Field Values
  - OPTION_IGNORE_LITERAL_SEQUENCES
```
static final String OPTION_IGNORE_LITERAL_SEQUENCES
```
    Ignore sequences of literals (e.g, 0,0,0,0...).
    
    See Also:
    
    Constant Field Values
  - IGNORE_USINGS
```
static final String IGNORE_USINGS
```
    Ignore using directives in C#. The default value is false.
    
    See Also:
    
    Constant Field Values
  - OPTION_SKIP_BLOCKS
```
static final String OPTION_SKIP_BLOCKS
```
    Enables or disabled skipping of blocks like a pre-processor. It is a boolean property. The default value is true.
    
    See Also:
    
    OPTION_SKIP_BLOCKS_PATTERN, Constant Field Values
  - OPTION_SKIP_BLOCKS_PATTERN
```
static final String OPTION_SKIP_BLOCKS_PATTERN
```
    Configures the pattern, to find the blocks to skip. It is a string property and contains of two parts, separated by |. The first part is the start pattern, the second part is the ending pattern. Default value is "#if 0|#endif".
    
    See Also:
    
    DEFAULT_SKIP_BLOCKS_PATTERN, Constant Field Values
  - DEFAULT_SKIP_BLOCKS_PATTERN
```
static final String DEFAULT_SKIP_BLOCKS_PATTERN
```
    See Also:
    
    Constant Field Values
- Method Detail
  - tokenize
```
void tokenize(SourceCode sourceCode,
              Tokens tokenEntries)
       throws IOException
```
    Throws:
    
    IOException

Interface Tokenizer

Field Summary

Method Summary

Field Detail

IGNORE_LITERALS

IGNORE_IDENTIFIERS

IGNORE_ANNOTATIONS

OPTION_IGNORE_LITERAL_SEQUENCES

IGNORE_USINGS

OPTION_SKIP_BLOCKS

OPTION_SKIP_BLOCKS_PATTERN

DEFAULT_SKIP_BLOCKS_PATTERN

Method Detail

tokenize