Package net.sourceforge.pmd.lang.ast.impl.javacc

Class JavaccToken

java.lang.Object
net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken
All Implemented Interfaces:
Comparable<JavaccToken>, GenericToken<JavaccToken>, Reportable
public class JavaccToken extends Object implements GenericToken<JavaccToken>
A generic token implementation for JavaCC parsers.

Largely has the same interface as the default generated token class. The main difference is that the position of the token is encoded as a start and end offset in the source file, instead of a (begin,end)x(line,column) 4-tuple. This offers two practical advantages:

  • It allows retrieving easily the underlying text of a node (just need to cut a substring of the file text). Other attributes like lines and column bounds can be derived as well - though this should not be done systematically because it's costlier.
  • It's a bit lighter. Token instances are one of the most numerous class in a typical PMD run and this may reduce GC pressure.

  • Field Details

    • EOF

      public static final int EOF
      Kind for EOF tokens.
      See Also:

    • IMPLICIT_TOKEN

      public static final int IMPLICIT_TOKEN
      Kind for implicit tokens. Negative because JavaCC only picks positive numbers for token kinds.
      See Also:

    • kind

      public final int kind
      An integer that describes the kind of this token. This numbering system is determined by JavaCCParser, and a table of these numbers is stored in the file ...Constants.java.

    • next

      public JavaccToken next
      A reference to the next regular (non-special) token from the input stream. If this is the last token from the input stream, or if the token manager has not read tokens beyond this one, this field is set to null. This is true only if this token is also a regular token. Otherwise, see below for a description of the contents of this field.

    • specialToken

      public JavaccToken specialToken
      This field is used to access special tokens that occur prior to this token, but after the immediately preceding regular (non-special) token. If there are no such special tokens, this field is set to null. When there are more than one such special token, this field refers to the last of these special tokens, which in turn refers to the next previous special token through its specialToken field, and so on until the first special token (whose specialToken field is null). The next fields of special tokens refer to other special tokens that immediately follow it (without an intervening regular token). If there is no such token, this field is null.

  • Constructor Details

    • JavaccToken

      public JavaccToken(int kind, Chars image, int startInclusive, int endExclusive, JavaccTokenDocument document)
      Builds a new token of the specified kind.
      Parameters:
      kind - Kind of token
      image - Image of the token (after translating escapes if any)
      startInclusive - Start character of the token in the text file (before translating escapes)
      endExclusive - End of the token in the text file (before translating escapes)
      document - Document owning the token

    • JavaccToken

      public JavaccToken(int kind, String image, int startInclusive, int endExclusive, JavaccTokenDocument document)
      Constructor with a String image (see the other ctor).

  • Method Details

    • getDocument

      public final JavaccTokenDocument getDocument()
      Returns the document owning this token.

    • isEof

      public boolean isEof()
      Description copied from interface: GenericToken
      Returns true if this token is an end-of-file token. This is the last token of token sequences that have been fully lexed.
      Specified by:
      isEof in interface GenericToken<JavaccToken>

    • getKind

      public int getKind()
      Description copied from interface: GenericToken
      Gets a unique integer representing the kind of token this is. The semantics of this kind depend on the language.

      The returned constants can be looked up in the language's "*ParserConstants", e.g. CppParserConstants or JavaParserConstants. These constants are considered internal API and may change at any time when the language's grammar is changed.

      Specified by:
      getKind in interface GenericToken<JavaccToken>

    • getNext

      public JavaccToken getNext()
      Description copied from interface: GenericToken
      Obtain the next generic token according to the input stream which generated the instance of this token.
      Specified by:
      getNext in interface GenericToken<JavaccToken>
      Returns:
      the next generic token if it exists; null if it does not exist

    • getPreviousComment

      public JavaccToken getPreviousComment()
      Description copied from interface: GenericToken
      Obtain a comment-type token which, according to the input stream which generated the instance of this token, precedes this instance token and succeeds the previous generic token (if there is any).
      Specified by:
      getPreviousComment in interface GenericToken<JavaccToken>
      Returns:
      the comment-type token if it exists; null if it does not exist

    • getImageCs

      public Chars getImageCs()
      Description copied from interface: GenericToken
      Returns the text of the token as a char sequence. This should be preferred when you can use eg StringUtils to do some processing, without having to create a string.
      Specified by:
      getImageCs in interface GenericToken<JavaccToken>

    • getImage

      public String getImage()
      Description copied from interface: GenericToken
      Returns the token's text as a string.
      Specified by:
      getImage in interface GenericToken<JavaccToken>

    • getText

      public Chars getText()
      Returns the original text of the token. The image may be normalized, e.g. for case-insensitive languages.
      Since:
      7.3.0

    • getRegion

      public final TextRegion getRegion()
      Description copied from interface: GenericToken
      Returns a text region with the coordinates of this token.
      Specified by:
      getRegion in interface GenericToken<JavaccToken>

    • getReportLocation

      public FileLocation getReportLocation()
      Description copied from interface: Reportable
      Returns the location at which this element should be reported.

      Use this instead of Node.getBeginColumn()/Node.getBeginLine(), etc.

      Specified by:
      getReportLocation in interface Reportable

    • isImplicit

      public boolean isImplicit()
      Description copied from interface: GenericToken
      Returns true if this token is implicit, ie was inserted artificially and has a zero-length image.
      Specified by:
      isImplicit in interface GenericToken<JavaccToken>

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • replaceImage

      public JavaccToken replaceImage(CharStream charStream)
      Returns a new token with the same kind as this one, whose image is replaced by the one marked on the char stream.
      Parameters:
      charStream - Char stream from which to start
      Returns:
      A new token

    • withKind

      public JavaccToken withKind(int newKind)
      Returns a new token with the given kind, and all other parameters identical to this one.
      Parameters:
      newKind - Char stream from which to start
      Returns:
      A new token

    • implicitBefore

      public static JavaccToken implicitBefore(JavaccToken next)
      Creates an implicit token, with zero length, that is linked to the given token as its special predecessor.
      Parameters:
      next - Token before which to insert the new token
      Returns:
      A new token

    • newImplicit

      public static JavaccToken newImplicit(int offset, JavaccTokenDocument document)
      Returns a new implicit token, positioned at the given offset.
      Parameters:
      offset - Offset of the token
      document - Document owning the token
      Returns:
      A new token