Class JavaccTokenDocument.TokenDocumentBehavior

  • Enclosing class:

    public static class JavaccTokenDocument.TokenDocumentBehavior
    extends Object
    Overridable configuration of a token document.
    • Constructor Detail

      • TokenDocumentBehavior

        public TokenDocumentBehavior​(List<String> tokenNames)
    • Method Detail

      • useMarkSuffix

        public boolean useMarkSuffix()
        Returns true if the lexer should accumulate the image of MORE tokens into the StringBuilder jjimage. This is useless in our current implementations, because the image of tokens can be cut out using text coordinates, so doesn't need to be put into a separate string. The default returns false, which makes CharStream.appendSuffix(StringBuilder, int) a noop.
      • describeKind

        public final @NonNull String describeKind​(int kind)
        Returns a string that describes the token kind.
        kind - Kind of token
        A descriptive string
      • describeKindImpl

        protected @Nullable String describeKindImpl​(int kind)
        Describe the given kind. If this returns a non-null value, then that's what describeKind(int) will use. Otherwise a default implementation is used.

        An implementation typically uses the JavaCC-generated array named <parser name>Constants.tokenImage. Remember to check the bounds of the array.

        kind - Kind of token
        A descriptive string, or null to use default
      • createToken

        public JavaccToken createToken​(JavaccTokenDocument self,
                                       int kind,
                                       CharStream cs,
                                       @Nullable String image)
        Creates a new token with the given kind. This is called back to by JavaCC-generated token managers (jjFillToken). Note that a created token is not guaranteed to end up in the final token chain.
        kind - Kind of the token
        cs - Char stream of the file. This can be used to get text coordinates and the image
        image - Shared instance of the image token. If this is non-null, then no call to CharStream.getTokenImage() should be issued.
        A new token