Class BaseTokenFilter<T extends GenericToken<T>>

    • Constructor Detail

      • BaseTokenFilter

        public BaseTokenFilter​(TokenManager<T> tokenManager)
        Creates a new BaseTokenFilter
        Parameters:
        tokenManager - The token manager from which to retrieve tokens to be filtered
    • Method Detail

      • analyzeToken

        protected void analyzeToken​(T currentToken)
        Extension point for subclasses to analyze all tokens (before filtering) and update internal status to decide on custom discard rules.
        Parameters:
        currentToken - The token to be analyzed
        See Also:
        isLanguageSpecificDiscarding()
      • analyzeTokens

        protected void analyzeTokens​(T currentToken,
                                     Iterable<T> remainingTokens)
        Extension point for subclasses to analyze all tokens (before filtering) and update internal status to decide on custom discard rules.
        Parameters:
        currentToken - The token to be analyzed
        remainingTokens - All upcoming tokens
        See Also:
        isLanguageSpecificDiscarding()
      • isLanguageSpecificDiscarding

        protected boolean isLanguageSpecificDiscarding()
        Extension point for subclasses to indicate tokens are to be filtered.
        Returns:
        True if tokens should be filtered, false otherwise
      • shouldStopProcessing

        protected boolean shouldStopProcessing​(T currentToken)
        Extension point for subclasses to indicate when to stop filtering tokens.
        Parameters:
        currentToken - The token to be analyzed
        Returns:
        True if the token filter has finished consuming all tokens, false otherwise