Interface GenericToken<T extends GenericToken<T>>

  • All Superinterfaces:
    Comparable<T>, Reportable
    All Known Implementing Classes:
    AntlrToken, JavaccToken

    public interface GenericToken<T extends GenericToken<T>>
    extends Comparable<T>, Reportable
    Represents a token, part of a token chain in a source file. Tokens are the individual "words" of a programming language, such as literals, identifiers, keywords, or comments. Tokens are produced by a lexer and are used by a parser implementation to build an AST Node. Tokens should generally not be manipulated in rules directly as they have little to no semantic information.
    • Method Detail

      • getNext

        T getNext()
        Obtain the next generic token according to the input stream which generated the instance of this token.
        Returns:
        the next generic token if it exists; null if it does not exist
      • getPreviousComment

        T getPreviousComment()
        Obtain a comment-type token which, according to the input stream which generated the instance of this token, precedes this instance token and succeeds the previous generic token (if there is any).
        Returns:
        the comment-type token if it exists; null if it does not exist
      • getImage

        default String getImage()
        Returns the token's text as a string.
      • getImageCs

        CharSequence getImageCs()
        Returns the text of the token as a char sequence. This should be preferred when you can use eg StringUtils to do some processing, without having to create a string.
      • imageEquals

        default boolean imageEquals​(CharSequence charSeq)
        Returns true if the image of this token equals the given charsequence. This does not create a string.
        Parameters:
        charSeq - A character sequence
      • getRegion

        TextRegion getRegion()
        Returns a text region with the coordinates of this token.
      • isEof

        boolean isEof()
        Returns true if this token is an end-of-file token. This is the last token of token sequences that have been fully lexed.
      • isImplicit

        default boolean isImplicit()
        Returns true if this token is implicit, ie was inserted artificially and has a zero-length image.
      • compareTo

        default int compareTo​(T o)
        This must return true if this token comes before the other token. If they start at the same index, then the smaller token comes before the other.
        Specified by:
        compareTo in interface Comparable<T extends GenericToken<T>>
      • range

        static <T extends GenericToken<T>> Iterable<T> range​(T from,
                                                             T to)
        Returns an iterator that enumerates all (non-special) tokens between the two tokens (bounds included).
        Parameters:
        from - First token to yield (inclusive)
        to - Last token to yield (inclusive)
        Returns:
        An iterator
        Throws:
        IllegalArgumentException - If the first token does not come before the other token
      • previousSpecials

        static <T extends GenericToken<T>> Iterable<T> previousSpecials​(T from)
        Returns an iterable that enumerates all special tokens belonging to the given token.
        Parameters:
        from - Token from which to start, note that the returned iterable does not contain that token
        Returns:
        An iterator, possibly empty, not containing the parameter
        Throws:
        NullPointerException - If the parameter s null
      • getKind

        @Experimental
        int getKind()
        Gets a unique integer representing the kind of token this is. The semantics of this kind depend on the language.

        Note: This is an experimental API.

        The returned constants can be looked up in the language's "*ParserConstants", e.g. CppParserConstants or JavaParserConstants. These constants are considered internal API and may change at any time when the language's grammar is changed.