Package net.sourceforge.pmd.cpd.impl

Class TokenizerBase<T extends GenericToken<T>>

java.lang.Object

net.sourceforge.pmd.cpd.impl.TokenizerBase<T>

All Implemented Interfaces:

Tokenizer

Direct Known Subclasses:

AntlrTokenizer, JavaCCTokenizer

public abstract class TokenizerBase<T extends GenericToken<T>>
extends Object
implements Tokenizer

Generic base class for a Tokenizer.

Constructor Summary

Constructors

Constructor	Description
TokenizerBase()

Method Summary

Modifier and Type	Method	Description
protected TokenManager<T>	filterTokenStream(TokenManager<T> tokenManager)
protected String	getImage(T token)
protected abstract TokenManager<T>	makeLexerImpl(TextDocument doc)
protected void	processToken(TokenFactory tokenEntries, T currentToken)
void	tokenize(TextDocument document, TokenFactory tokens)	Tokenize the source code and record tokens using the provided token factory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerBase

public TokenizerBase()

Method Detail

makeLexerImpl

protected abstract TokenManager<T> makeLexerImpl(TextDocument doc)
                                          throws IOException

Throws:

IOException

filterTokenStream

protected TokenManager<T> filterTokenStream(TokenManager<T> tokenManager)

processToken

protected void processToken(TokenFactory tokenEntries,
                            T currentToken)

getImage

protected String getImage(T token)

tokenize

public final void tokenize(TextDocument document,
                           TokenFactory tokens)
                    throws IOException

Description copied from interface: Tokenizer

Tokenize the source code and record tokens using the provided token factory.

Specified by:

tokenize in interface Tokenizer

Throws:

IOException