Package adams.flow.transformer.tokenizer

Class AbstractTokenizer

  • All Implemented Interfaces:
    adams.core.Destroyable, adams.core.GlobalInfoSupporter, adams.core.logging.LoggingLevelHandler, adams.core.logging.LoggingSupporter, adams.core.option.OptionHandler, adams.core.SizeOfHandler, Serializable
    Direct Known Subclasses:
    StanfordPTBTokenizer, Twokenize, WekaTokenizer
    public abstract class AbstractTokenizer
extends adams.core.option.AbstractOptionHandler
    Ancestor for classes that split strings into words ("tokenize").
    Version:
    $Revision: 11956 $
    Author:
    fracpete (fracpete at waikato dot ac dot nz)
    See Also:
    Serialized Form

    • Field Summary

      • Fields inherited from class adams.core.option.AbstractOptionHandler

        m_OptionManager

      • Fields inherited from class adams.core.logging.LoggingObject

        m_Logger, m_LoggingIsEnabled, m_LoggingLevel

    • Method Summary

      All Methods Instance Methods Abstract Methods Concrete Methods 
      Modifier and Type Method Description
      protected void check​(String str)
      Checks the string.
      protected abstract List<String> doTokenize​(String str)
      Performs the actual tokenization.
      List<String> tokenize​(String str)
      Tokenizes the given string into words.

      • Methods inherited from class adams.core.option.AbstractOptionHandler

        cleanUpOptions, defineOptions, destroy, finishInit, getDefaultLoggingLevel, getOptionManager, globalInfo, initialize, loggingLevelTipText, newOptionManager, reset, setLoggingLevel, toCommandLine, toString

      • Methods inherited from class adams.core.logging.LoggingObject

        configureLogger, getLogger, getLoggingLevel, initializeLogging, isLoggingEnabled, sizeOf

      • Methods inherited from interface adams.core.logging.LoggingLevelHandler

        getLoggingLevel

    • Constructor Detail

      • AbstractTokenizer

        public AbstractTokenizer()

    • Method Detail

      • check

        protected void check​(String str)
        Checks the string.

        Default implementation only checks whether a string was provided.
        Parameters:
        str - the string to check

      • doTokenize

        protected abstract List<String> doTokenize​(String str)
        Performs the actual tokenization.
        Parameters:
        str - the string to tokenize
        Returns:
        the list of words

      • tokenize

        public List<String> tokenize​(String str)
        Tokenizes the given string into words.
        Parameters:
        str - the string to process
        Returns:
        the generated words