Package weka.core.tokenizers
Class MultiTokenizer
- java.lang.Object
-
- weka.core.tokenizers.Tokenizer
-
- weka.core.tokenizers.MultiTokenizer
-
- All Implemented Interfaces:
Serializable
,Enumeration<String>
,weka.core.OptionHandler
,weka.core.RevisionHandler
public class MultiTokenizer extends weka.core.tokenizers.Tokenizer
Combines the tokens of several tokenizers, skipping duplicate tokens.- Version:
- $Revision$
- Author:
- FracPete (fracpete at waikato dot ac dot nz)
- See Also:
- Serialized Form
-
-
Constructor Summary
Constructors Constructor Description MultiTokenizer()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected weka.core.tokenizers.Tokenizer[]
getDefaultTokenizers()
Returns the default token tokenizers.String[]
getOptions()
Gets the current option settings for the OptionHandler.String
getRevision()
Returns the revision string.weka.core.tokenizers.Tokenizer[]
getTokenizers()
Returns the tokenizers to use.String
globalInfo()
Returns a string describing the stemmerboolean
hasMoreElements()
Tests if this enumeration contains more elements.Enumeration
listOptions()
Returns an enumeration describing the available options.String
nextElement()
Returns the next element of this enumeration if this enumeration object has at least one more element to provide.void
setOptions(String[] options)
Sets the OptionHandler's options using the given list.void
setTokenizers(weka.core.tokenizers.Tokenizer[] value)
Sets the tokenizers to use.void
tokenize(String s)
Sets the string to tokenize.String
tokenizersTipText()
Returns the tip text for this property.-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface java.util.Enumeration
asIterator
-
-
-
-
Field Detail
-
TOKENIZER
public static final String TOKENIZER
- See Also:
- Constant Field Values
-
m_Tokenizers
protected weka.core.tokenizers.Tokenizer[] m_Tokenizers
the tokenizers to use.
-
-
Method Detail
-
globalInfo
public String globalInfo()
Returns a string describing the stemmer- Specified by:
globalInfo
in classweka.core.tokenizers.Tokenizer
- Returns:
- a description suitable for displaying in the explorer/experimenter gui
-
listOptions
public Enumeration listOptions()
Returns an enumeration describing the available options.- Specified by:
listOptions
in interfaceweka.core.OptionHandler
- Overrides:
listOptions
in classweka.core.tokenizers.Tokenizer
- Returns:
- an enumeration of all the available options.
-
setOptions
public void setOptions(String[] options) throws Exception
Sets the OptionHandler's options using the given list. All options will be set (or reset) during this call (i.e. incremental setting of options is not possible).- Specified by:
setOptions
in interfaceweka.core.OptionHandler
- Overrides:
setOptions
in classweka.core.tokenizers.Tokenizer
- Parameters:
options
- the list of options as an array of strings- Throws:
Exception
- if an option is not supported
-
getOptions
public String[] getOptions()
Gets the current option settings for the OptionHandler.- Specified by:
getOptions
in interfaceweka.core.OptionHandler
- Overrides:
getOptions
in classweka.core.tokenizers.Tokenizer
- Returns:
- the list of current option settings as an array of strings
-
getDefaultTokenizers
protected weka.core.tokenizers.Tokenizer[] getDefaultTokenizers()
Returns the default token tokenizers.- Returns:
- the default
-
setTokenizers
public void setTokenizers(weka.core.tokenizers.Tokenizer[] value)
Sets the tokenizers to use.- Parameters:
value
- the tokenizers
-
getTokenizers
public weka.core.tokenizers.Tokenizer[] getTokenizers()
Returns the tokenizers to use.- Returns:
- the tokenizers
-
tokenizersTipText
public String tokenizersTipText()
Returns the tip text for this property.- Returns:
- tip text for this property suitable for displaying in the GUI or for listing the options.
-
hasMoreElements
public boolean hasMoreElements()
Tests if this enumeration contains more elements.- Specified by:
hasMoreElements
in interfaceEnumeration<String>
- Specified by:
hasMoreElements
in classweka.core.tokenizers.Tokenizer
- Returns:
- true if and only if this enumeration object contains at least one more element to provide; false otherwise.
-
nextElement
public String nextElement()
Returns the next element of this enumeration if this enumeration object has at least one more element to provide.- Specified by:
nextElement
in interfaceEnumeration<String>
- Specified by:
nextElement
in classweka.core.tokenizers.Tokenizer
- Returns:
- the next element of this enumeration.
-
tokenize
public void tokenize(String s)
Sets the string to tokenize. Tokenization happens immediately.- Specified by:
tokenize
in classweka.core.tokenizers.Tokenizer
- Parameters:
s
- the string to tokenize
-
getRevision
public String getRevision()
Returns the revision string.- Returns:
- the revision
-
-