Package weka.core.tokenizers

Class MultiTokenizer

java.lang.Object

weka.core.tokenizers.Tokenizer

weka.core.tokenizers.MultiTokenizer

All Implemented Interfaces:

Serializable, Enumeration<String>, weka.core.OptionHandler, weka.core.RevisionHandler

public class MultiTokenizer
extends weka.core.tokenizers.Tokenizer

Combines the tokens of several tokenizers, skipping duplicate tokens.

Version:

$Revision$

Author:

FracPete (fracpete at waikato dot ac dot nz)

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
protected weka.core.tokenizers.Tokenizer[]	m_Tokenizers	the tokenizers to use.
protected List<String>	m_Tokens	the collected tokens.
static String	TOKENIZER

Constructor Summary

Constructors

Constructor	Description
MultiTokenizer()

Method Summary

Modifier and Type	Method	Description
protected weka.core.tokenizers.Tokenizer[]	getDefaultTokenizers()	Returns the default token tokenizers.
String[]	getOptions()	Gets the current option settings for the OptionHandler.
String	getRevision()	Returns the revision string.
weka.core.tokenizers.Tokenizer[]	getTokenizers()	Returns the tokenizers to use.
String	globalInfo()	Returns a string describing the stemmer
boolean	hasMoreElements()	Tests if this enumeration contains more elements.
Enumeration	listOptions()	Returns an enumeration describing the available options.
String	nextElement()	Returns the next element of this enumeration if this enumeration object has at least one more element to provide.
void	setOptions(String[] options)	Sets the OptionHandler's options using the given list.
void	setTokenizers(weka.core.tokenizers.Tokenizer[] value)	Sets the tokenizers to use.
void	tokenize(String s)	Sets the string to tokenize.
String	tokenizersTipText()	Returns the tip text for this property.

Methods inherited from class weka.core.tokenizers.Tokenizer

runTokenizer, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Enumeration

asIterator

Field Detail

TOKENIZER

public static final String TOKENIZER

See Also:

Constant Field Values

m_Tokenizers

protected weka.core.tokenizers.Tokenizer[] m_Tokenizers

the tokenizers to use.

m_Tokens

protected List<String> m_Tokens

the collected tokens.

Constructor Detail

MultiTokenizer

public MultiTokenizer()

Method Detail

globalInfo

public String globalInfo()

Returns a string describing the stemmer

Specified by:

globalInfo in class weka.core.tokenizers.Tokenizer

Returns:

a description suitable for displaying in the explorer/experimenter gui

listOptions

public Enumeration listOptions()

Returns an enumeration describing the available options.

Specified by:

listOptions in interface weka.core.OptionHandler

Overrides:

listOptions in class weka.core.tokenizers.Tokenizer

Returns:

an enumeration of all the available options.

setOptions

public void setOptions(String[] options)
                throws Exception

Sets the OptionHandler's options using the given list. All options will be set (or reset) during this call (i.e. incremental setting of options is not possible).

Specified by:

setOptions in interface weka.core.OptionHandler

Overrides:

setOptions in class weka.core.tokenizers.Tokenizer

Parameters:

options - the list of options as an array of strings

Throws:

Exception - if an option is not supported

getOptions

public String[] getOptions()

Gets the current option settings for the OptionHandler.

Specified by:

getOptions in interface weka.core.OptionHandler

Overrides:

getOptions in class weka.core.tokenizers.Tokenizer

Returns:

the list of current option settings as an array of strings

getDefaultTokenizers

protected weka.core.tokenizers.Tokenizer[] getDefaultTokenizers()

Returns the default token tokenizers.

Returns:

the default

setTokenizers

public void setTokenizers(weka.core.tokenizers.Tokenizer[] value)

Sets the tokenizers to use.

Parameters:

value - the tokenizers

getTokenizers

public weka.core.tokenizers.Tokenizer[] getTokenizers()

Returns the tokenizers to use.

Returns:

the tokenizers

tokenizersTipText

public String tokenizersTipText()

Returns the tip text for this property.

Returns:

tip text for this property suitable for displaying in the GUI or for listing the options.

hasMoreElements

public boolean hasMoreElements()

Tests if this enumeration contains more elements.

Specified by:

hasMoreElements in interface Enumeration<String>

Specified by:

hasMoreElements in class weka.core.tokenizers.Tokenizer

Returns:

true if and only if this enumeration object contains at least one more element to provide; false otherwise.

nextElement

public String nextElement()

Returns the next element of this enumeration if this enumeration object has at least one more element to provide.

Specified by:

nextElement in interface Enumeration<String>

Specified by:

nextElement in class weka.core.tokenizers.Tokenizer

Returns:

the next element of this enumeration.

tokenize

public void tokenize(String s)

Sets the string to tokenize. Tokenization happens immediately.

Specified by:

tokenize in class weka.core.tokenizers.Tokenizer

Parameters:

s - the string to tokenize

getRevision

public String getRevision()

Returns the revision string.

Returns:

the revision