Package net.sourceforge.pmd.lang.ast

Interface GenericToken<T extends GenericToken<T>>

  • All Superinterfaces:
    Comparable<T>, Reportable
    All Known Implementing Classes:
    AntlrToken, JavaccToken
    public interface GenericToken<T extends GenericToken<T>>
extends Comparable<T>, Reportable
    Represents a token, part of a token chain in a source file. Tokens are the individual "words" of a programming language, such as literals, identifiers, keywords, or comments. Tokens are produced by a lexer and are used by a parser implementation to build an AST Node. Tokens should generally not be manipulated in rules directly as they have little to no semantic information.

    • Method Summary

      All Methods Static Methods Instance Methods Abstract Methods Default Methods 
      Modifier and Type Method Description
      default int compareTo​(T o)
      This must return true if this token comes before the other token.
      default String getImage()
      Returns the token's text as a string.
      CharSequence getImageCs()
      Returns the text of the token as a char sequence.
      int getKind()
      Gets a unique integer representing the kind of token this is.
      T getNext()
      Obtain the next generic token according to the input stream which generated the instance of this token.
      T getPreviousComment()
      Obtain a comment-type token which, according to the input stream which generated the instance of this token, precedes this instance token and succeeds the previous generic token (if there is any).
      TextRegion getRegion()
      Returns a text region with the coordinates of this token.
      default boolean imageEquals​(CharSequence charSeq)
      Returns true if the image of this token equals the given charsequence.
      boolean isEof()
      Returns true if this token is an end-of-file token.
      default boolean isImplicit()
      Returns true if this token is implicit, ie was inserted artificially and has a zero-length image.
      static <T extends GenericToken<T>>
      Iterable<T>      		 previousSpecials​(T from)
      Returns an iterable that enumerates all special tokens belonging to the given token.
      static <T extends GenericToken<T>>
      Iterable<T>      		 range​(T from, T to)
      Returns an iterator that enumerates all (non-special) tokens between the two tokens (bounds included).
      static <T extends GenericToken<T>>
      Stream<T>      		 streamRange​(T from, T to)
      Returns a stream corresponding to range(GenericToken, GenericToken).

    • Method Detail

      • getNext

        T getNext()
        Obtain the next generic token according to the input stream which generated the instance of this token.
        Returns:
        the next generic token if it exists; null if it does not exist

      • getPreviousComment

        T getPreviousComment()
        Obtain a comment-type token which, according to the input stream which generated the instance of this token, precedes this instance token and succeeds the previous generic token (if there is any).
        Returns:
        the comment-type token if it exists; null if it does not exist

      • getImage

        default String getImage()
        Returns the token's text as a string.

      • getImageCs

        CharSequence getImageCs()
        Returns the text of the token as a char sequence. This should be preferred when you can use eg StringUtils to do some processing, without having to create a string.

      • imageEquals

        default boolean imageEquals​(CharSequence charSeq)
        Returns true if the image of this token equals the given charsequence. This does not create a string.
        Parameters:
        charSeq - A character sequence

      • getRegion

        TextRegion getRegion()
        Returns a text region with the coordinates of this token.

      • isEof

        boolean isEof()
        Returns true if this token is an end-of-file token. This is the last token of token sequences that have been fully lexed.

      • isImplicit

        default boolean isImplicit()
        Returns true if this token is implicit, ie was inserted artificially and has a zero-length image.

      • compareTo

        default int compareTo​(T o)
        This must return true if this token comes before the other token. If they start at the same index, then the smaller token comes before the other.
        Specified by:
        compareTo in interface Comparable<T extends GenericToken<T>>

      • range

        static <T extends GenericToken<T>> Iterable<T> range​(T from,
                                                     T to)
        Returns an iterator that enumerates all (non-special) tokens between the two tokens (bounds included).
        Parameters:
        from - First token to yield (inclusive)
        to - Last token to yield (inclusive)
        Returns:
        An iterator
        Throws:
        IllegalArgumentException - If the first token does not come before the other token

      • previousSpecials

        static <T extends GenericToken<T>> Iterable<T> previousSpecials​(T from)
        Returns an iterable that enumerates all special tokens belonging to the given token.
        Parameters:
        from - Token from which to start, note that the returned iterable does not contain that token
        Returns:
        An iterator, possibly empty, not containing the parameter
        Throws:
        NullPointerException - If the parameter s null

      • getKind

        int getKind()
        Gets a unique integer representing the kind of token this is. The semantics of this kind depend on the language.

        The returned constants can be looked up in the language's "*ParserConstants", e.g. CppParserConstants or JavaParserConstants. These constants are considered internal API and may change at any time when the language's grammar is changed.