Class MultiTokenizer

  • All Implemented Interfaces:
    Serializable, Enumeration<String>, weka.core.OptionHandler, weka.core.RevisionHandler

    public class MultiTokenizer
    extends weka.core.tokenizers.Tokenizer
    Combines the tokens of several tokenizers, skipping duplicate tokens.
    Version:
    $Revision$
    Author:
    FracPete (fracpete at waikato dot ac dot nz)
    See Also:
    Serialized Form
    • Field Detail

      • m_Tokenizers

        protected weka.core.tokenizers.Tokenizer[] m_Tokenizers
        the tokenizers to use.
      • m_Tokens

        protected List<String> m_Tokens
        the collected tokens.
    • Constructor Detail

      • MultiTokenizer

        public MultiTokenizer()
    • Method Detail

      • globalInfo

        public String globalInfo()
        Returns a string describing the stemmer
        Specified by:
        globalInfo in class weka.core.tokenizers.Tokenizer
        Returns:
        a description suitable for displaying in the explorer/experimenter gui
      • listOptions

        public Enumeration listOptions()
        Returns an enumeration describing the available options.
        Specified by:
        listOptions in interface weka.core.OptionHandler
        Overrides:
        listOptions in class weka.core.tokenizers.Tokenizer
        Returns:
        an enumeration of all the available options.
      • setOptions

        public void setOptions​(String[] options)
                        throws Exception
        Sets the OptionHandler's options using the given list. All options will be set (or reset) during this call (i.e. incremental setting of options is not possible).
        Specified by:
        setOptions in interface weka.core.OptionHandler
        Overrides:
        setOptions in class weka.core.tokenizers.Tokenizer
        Parameters:
        options - the list of options as an array of strings
        Throws:
        Exception - if an option is not supported
      • getOptions

        public String[] getOptions()
        Gets the current option settings for the OptionHandler.
        Specified by:
        getOptions in interface weka.core.OptionHandler
        Overrides:
        getOptions in class weka.core.tokenizers.Tokenizer
        Returns:
        the list of current option settings as an array of strings
      • getDefaultTokenizers

        protected weka.core.tokenizers.Tokenizer[] getDefaultTokenizers()
        Returns the default token tokenizers.
        Returns:
        the default
      • setTokenizers

        public void setTokenizers​(weka.core.tokenizers.Tokenizer[] value)
        Sets the tokenizers to use.
        Parameters:
        value - the tokenizers
      • getTokenizers

        public weka.core.tokenizers.Tokenizer[] getTokenizers()
        Returns the tokenizers to use.
        Returns:
        the tokenizers
      • tokenizersTipText

        public String tokenizersTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.
      • hasMoreElements

        public boolean hasMoreElements()
        Tests if this enumeration contains more elements.
        Specified by:
        hasMoreElements in interface Enumeration<String>
        Specified by:
        hasMoreElements in class weka.core.tokenizers.Tokenizer
        Returns:
        true if and only if this enumeration object contains at least one more element to provide; false otherwise.
      • nextElement

        public String nextElement()
        Returns the next element of this enumeration if this enumeration object has at least one more element to provide.
        Specified by:
        nextElement in interface Enumeration<String>
        Specified by:
        nextElement in class weka.core.tokenizers.Tokenizer
        Returns:
        the next element of this enumeration.
      • tokenize

        public void tokenize​(String s)
        Sets the string to tokenize. Tokenization happens immediately.
        Specified by:
        tokenize in class weka.core.tokenizers.Tokenizer
        Parameters:
        s - the string to tokenize
      • getRevision

        public String getRevision()
        Returns the revision string.
        Returns:
        the revision