Package adams.flow.transformer.tokenizer

Class Twokenize

  • All Implemented Interfaces:
    adams.core.Destroyable, adams.core.GlobalInfoSupporter, adams.core.logging.LoggingLevelHandler, adams.core.logging.LoggingSupporter, adams.core.option.OptionHandler, adams.core.SizeOfHandler, Serializable
    public class Twokenize
extends AbstractTokenizer
    Uses TweetNLP's Twokenize.

    For more details on the tokenizer see:
    https://github.com/brendano/ark-tweet-nlp/blob/master/src/cmu/arktweetnlp/Twokenize.java 

    -logging-level <OFF|SEVERE|WARNING|INFO|CONFIG|FINE|FINER|FINEST> (property: loggingLevel)
    The logging level for outputting errors and debugging output.
    default: WARNING
    Version:
    $Revision: 10826 $
    Author:
    fracpete (fracpete at waikato dot ac dot nz)
    See Also:
    Serialized Form

    • Field Summary

      • Fields inherited from class adams.core.option.AbstractOptionHandler

        m_OptionManager

      • Fields inherited from class adams.core.logging.LoggingObject

        m_Logger, m_LoggingIsEnabled, m_LoggingLevel

    • Constructor Summary

      Constructors 
      Constructor Description
      Twokenize()  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected List<String> doTokenize​(String str)
      Performs the actual tokenization.
      String globalInfo()
      Returns a string describing the object.

      • Methods inherited from class adams.core.option.AbstractOptionHandler

        cleanUpOptions, defineOptions, destroy, finishInit, getDefaultLoggingLevel, getOptionManager, initialize, loggingLevelTipText, newOptionManager, reset, setLoggingLevel, toCommandLine, toString

      • Methods inherited from class adams.core.logging.LoggingObject

        configureLogger, getLogger, getLoggingLevel, initializeLogging, isLoggingEnabled, sizeOf

      • Methods inherited from interface adams.core.logging.LoggingLevelHandler

        getLoggingLevel

    • Constructor Detail

      • Twokenize

        public Twokenize()

    • Method Detail

      • globalInfo

        public String globalInfo()
        Returns a string describing the object.
        Specified by:
        globalInfo in interface adams.core.GlobalInfoSupporter
        Specified by:
        globalInfo in class adams.core.option.AbstractOptionHandler
        Returns:
        a description suitable for displaying in the gui

      • doTokenize

        protected List<String> doTokenize​(String str)
        Performs the actual tokenization.
        Specified by:
        doTokenize in class AbstractTokenizer
        Parameters:
        str - the string to tokenize
        Returns:
        the list of sentence words