Package net.sourceforge.pmd.lang.ast.impl.javacc

Class JavaccToken

  • All Implemented Interfaces:
    Comparable<JavaccToken>, GenericToken<JavaccToken>, Reportable
    public class JavaccToken
extends Object
implements GenericToken<JavaccToken>
    A generic token implementation for JavaCC parsers.

    Largely has the same interface as the default generated token class. The main difference is that the position of the token is encoded as a start and end offset in the source file, instead of a (begin,end)x(line,column) 4-tuple. This offers two practical advantages:

    • It allows retrieving easily the underlying text of a node (just need to cut a substring of the file text). Other attributes like lines and column bounds can be derived as well - though this should not be done systematically because it's costlier.
    • It's a bit lighter. Token instances are one of the most numerous class in a typical PMD run and this may reduce GC pressure.

    • Field Detail

      • IMPLICIT_TOKEN

        public static final int IMPLICIT_TOKEN
        Kind for implicit tokens. Negative because JavaCC only picks positive numbers for token kinds.
        See Also:
        Constant Field Values

      • kind

        public final int kind
        An integer that describes the kind of this token. This numbering system is determined by JavaCCParser, and a table of these numbers is stored in the file ...Constants.java.

      • next

        public JavaccToken next
        A reference to the next regular (non-special) token from the input stream. If this is the last token from the input stream, or if the token manager has not read tokens beyond this one, this field is set to null. This is true only if this token is also a regular token. Otherwise, see below for a description of the contents of this field.

      • specialToken

        public JavaccToken specialToken
        This field is used to access special tokens that occur prior to this token, but after the immediately preceding regular (non-special) token. If there are no such special tokens, this field is set to null. When there are more than one such special token, this field refers to the last of these special tokens, which in turn refers to the next previous special token through its specialToken field, and so on until the first special token (whose specialToken field is null). The next fields of special tokens refer to other special tokens that immediately follow it (without an intervening regular token). If there is no such token, this field is null.

    • Constructor Detail

      • JavaccToken

        public JavaccToken​(int kind,
                   Chars image,
                   int startInclusive,
                   int endExclusive,
                   JavaccTokenDocument document)
        Builds a new token of the specified kind.
        Parameters:
        kind - Kind of token
        image - Image of the token (after translating escapes if any)
        startInclusive - Start character of the token in the text file (before translating escapes)
        endExclusive - End of the token in the text file (before translating escapes)
        document - Document owning the token

    • Method Detail

      • getDocument

        public final JavaccTokenDocument getDocument()
        Returns the document owning this token.

      • isEof

        public boolean isEof()
        Description copied from interface: GenericToken
        Returns true if this token is an end-of-file token. This is the last token of token sequences that have been fully lexed.
        Specified by:
        isEof in interface GenericToken<JavaccToken>

      • getKind

        public int getKind()
        Description copied from interface: GenericToken
        Gets a unique integer representing the kind of token this is. The semantics of this kind depend on the language.

        Note: This is an experimental API.

        The returned constants can be looked up in the language's "*ParserConstants", e.g. CppParserConstants or JavaParserConstants. These constants are considered internal API and may change at any time when the language's grammar is changed.

        Specified by:
        getKind in interface GenericToken<JavaccToken>

      • getNext

        public JavaccToken getNext()
        Description copied from interface: GenericToken
        Obtain the next generic token according to the input stream which generated the instance of this token.
        Specified by:
        getNext in interface GenericToken<JavaccToken>
        Returns:
        the next generic token if it exists; null if it does not exist

      • getPreviousComment

        public JavaccToken getPreviousComment()
        Description copied from interface: GenericToken
        Obtain a comment-type token which, according to the input stream which generated the instance of this token, precedes this instance token and succeeds the previous generic token (if there is any).
        Specified by:
        getPreviousComment in interface GenericToken<JavaccToken>
        Returns:
        the comment-type token if it exists; null if it does not exist

      • getImageCs

        public Chars getImageCs()
        Description copied from interface: GenericToken
        Returns the text of the token as a char sequence. This should be preferred when you can use eg StringUtils to do some processing, without having to create a string.
        Specified by:
        getImageCs in interface GenericToken<JavaccToken>

      • isImplicit

        public boolean isImplicit()
        Description copied from interface: GenericToken
        Returns true if this token is implicit, ie was inserted artificially and has a zero-length image.
        Specified by:
        isImplicit in interface GenericToken<JavaccToken>

      • replaceImage

        public JavaccToken replaceImage​(CharStream charStream)
        Returns a new token with the same kind as this one, whose image is replaced by the one marked on the char stream.
        Parameters:
        charStream - Char stream from which to start
        Returns:
        A new token

      • withKind

        public JavaccToken withKind​(int newKind)
        Returns a new token with the given kind, and all other parameters identical to this one.
        Parameters:
        newKind - Char stream from which to start
        Returns:
        A new token

      • implicitBefore

        public static JavaccToken implicitBefore​(JavaccToken next)
        Creates an implicit token, with zero length, that is linked to the given token as its special predecessor.
        Parameters:
        next - Token before which to insert the new token
        Returns:
        A new token

      • newImplicit

        public static JavaccToken newImplicit​(int offset,
                                      JavaccTokenDocument document)
        Returns a new implicit token, positioned at the given offset.
        Parameters:
        offset - Offset of the token
        document - Document owning the token
        Returns:
        A new token