Package weka.core.tokenizers
-
Class Summary Class Description MultiTokenizer Combines the tokens of several tokenizers, skipping duplicate tokens.PreCleanedTokenizer Allows the cleaning of tokens before actual tokenization.
| Class | Description |
|---|---|
| MultiTokenizer |
Combines the tokens of several tokenizers, skipping duplicate tokens.
|
| PreCleanedTokenizer |
Allows the cleaning of tokens before actual tokenization.
|