Uses of Class
weka.core.tokenizers.Tokenizer

Packages that use Tokenizer

weka.core.tokenizers
weka.filters.unsupervised.attribute

Uses of Tokenizer in weka.core.tokenizers

Subclasses of Tokenizer in weka.core.tokenizers

class	AlphabeticTokenizer Alphabetic string tokenizer, tokens are to be formed only from contiguous alphabetic sequences.
class	CharacterDelimitedTokenizer Abstract superclass for tokenizers that take characters as delimiters.
class	NGramTokenizer Splits a string into an n-gram with min and max grams.
class	WordTokenizer A simple tokenizer that is using the java.util.StringTokenizer class to tokenize the strings.

Methods in weka.core.tokenizers with parameters of type Tokenizer

static void	Tokenizer.runTokenizer(Tokenizer tokenizer, String[] options) initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array.
static String[]	Tokenizer.tokenize(Tokenizer tokenizer, String[] options) initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array.

Uses of Tokenizer in weka.filters.unsupervised.attribute

Methods in weka.filters.unsupervised.attribute that return Tokenizer

Tokenizer

StringToWordVector.getTokenizer() Returns the current tokenizer algorithm.

Methods in weka.filters.unsupervised.attribute with parameters of type Tokenizer

void	StringToWordVector.setTokenizer(Tokenizer value) the tokenizer algorithm to use.