Package weka.core.tokenizers

Class TwitterNLPTokenizer

  • All Implemented Interfaces:
    Serializable, Enumeration<String>, weka.core.OptionHandler, weka.core.RevisionHandler
    public class TwitterNLPTokenizer
extends weka.core.tokenizers.Tokenizer
    Tokenizer using TweetNLP's Twokenize. Taken from here
    Version:
    $Revision$
    Author:
    Felipe Bravo
    See Also:
    Serialized Form

    • Field Detail

      • m_TokenIterator

        protected transient Iterator<String> m_TokenIterator
        the iterator for the tokens.

      • m_UseLowerCase

        protected boolean m_UseLowerCase
        whether to lower-case the tweet.

      • m_Cleaner

        protected weka.core.tokenizers.cleaners.TokenCleaner m_Cleaner
        the cleaner to use.

    • Constructor Detail

      • TwitterNLPTokenizer

        public TwitterNLPTokenizer()

    • Method Detail

      • globalInfo

        public String globalInfo()
        Returns a string describing the tokenizer.
        Specified by:
        globalInfo in class weka.core.tokenizers.Tokenizer
        Returns:
        a description suitable for displaying in the explorer/experimenter gui

      • listOptions

        public Enumeration listOptions()
        Returns an enumeration describing the available options.
        Specified by:
        listOptions in interface weka.core.OptionHandler
        Overrides:
        listOptions in class weka.core.tokenizers.Tokenizer
        Returns:
        an enumeration of all the available options.

      • setOptions

        public void setOptions​(String[] options)
                throws Exception
        Sets the OptionHandler's options using the given list. All options will be set (or reset) during this call (i.e. incremental setting of options is not possible).
        Specified by:
        setOptions in interface weka.core.OptionHandler
        Overrides:
        setOptions in class weka.core.tokenizers.Tokenizer
        Parameters:
        options - the list of options as an array of strings
        Throws:
        Exception - if an option is not supported

      • getOptions

        public String[] getOptions()
        Gets the current option settings for the OptionHandler.
        Specified by:
        getOptions in interface weka.core.OptionHandler
        Overrides:
        getOptions in class weka.core.tokenizers.Tokenizer
        Returns:
        the list of current option settings as an array of strings

      • setUseLowerCase

        public void setUseLowerCase​(boolean value)
        Sets whether to use lower case.
        Parameters:
        value - true if to use lower case

      • getUseLowerCase

        public boolean getUseLowerCase()
        Returns whether to use lower case.
        Returns:
        true if to use lower case

      • useLowerCaseTipText

        public String useLowerCaseTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • getDefaultCleaner

        protected weka.core.tokenizers.cleaners.TokenCleaner getDefaultCleaner()
        Returns the default token cleaner.
        Returns:
        the default

      • setCleaner

        public void setCleaner​(weka.core.tokenizers.cleaners.TokenCleaner value)
        Sets the token cleaner to use.
        Parameters:
        value - the cleaner

      • getCleaner

        public weka.core.tokenizers.cleaners.TokenCleaner getCleaner()
        Returns the token cleaner to use.
        Returns:
        the cleaner

      • cleanerTipText

        public String cleanerTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • hasMoreElements

        public boolean hasMoreElements()
        Tests if this enumeration contains more elements.
        Specified by:
        hasMoreElements in interface Enumeration<String>
        Specified by:
        hasMoreElements in class weka.core.tokenizers.Tokenizer
        Returns:
        true if and only if this enumeration object contains at least one more element to provide; false otherwise.

      • nextElement

        public String nextElement()
        Returns the next element of this enumeration if this enumeration object has at least one more element to provide.
        Specified by:
        nextElement in interface Enumeration<String>
        Specified by:
        nextElement in class weka.core.tokenizers.Tokenizer
        Returns:
        the next element of this enumeration.

      • tokenize

        public void tokenize​(String s)
        Sets the string to tokenize. Tokenization happens immediately.
        Specified by:
        tokenize in class weka.core.tokenizers.Tokenizer
        Parameters:
        s - the string to tokenize

      • getRevision

        public String getRevision()
        Returns the revision string.
        Returns:
        the revision

      • main

        public static void main​(String[] args)
        Runs the tokenizer with the given options and strings to tokenize. The tokens are printed to stdout.
        Parameters:
        args - the commandline options and strings to tokenize