Twokenize

```
public class Twokenize
extends java.lang.Object
```
Twokenize -- a tokenizer designed for Twitter text in English and some other European languages. This tokenizer code has gone through a long history: (1) Brendan O'Connor wrote original version in Python, http://github.com/brendano/tweetmotif TweetMotif: Exploratory Search and Topic Summarization for Twitter. Brendan O'Connor, Michel Krieger, and David Ahn. ICWSM-2010 (demo track), http://brenocon.com/oconnor_krieger_ahn.icwsm2010.tweetmotif.pdf (2a) Kevin Gimpel and Daniel Mills modified it for POS tagging for the CMU ARK Twitter POS Tagger (2b) Jason Baldridge and David Snyder ported it to Scala (3) Brendan bugfixed the Scala port and merged with POS-specific changes for the CMU ARK Twitter POS Tagger (4) Tobi Owoputi ported it back to Java and added many improvements (2012-06) Current home is http://github.com/brendano/ark-tweet-nlp and http://www.ark.cs.cmu.edu/TweetNLP There have been at least 2 other Java ports, but they are not in the lineage for the code here.

Constructors
Constructor and Description
`Twokenize()`

Methods
Modifier and Type	Method and Description
`static java.lang.String`	`normalizeTextForTagger(java.lang.String text)` Twitter text comes HTML-escaped, so unescape it.
`static java.lang.String`	`OR(java.lang.String... parts)`
`static java.lang.String`	`splitEdgePunct(java.lang.String input)`
`static java.lang.String`	`squeezeWhitespace(java.lang.String input)` "foo bar " => "foo bar"
`static java.util.List<java.lang.String>`	`tokenize(java.lang.String text)` Assume 'text' has no HTML escaping.
`static java.util.List<java.lang.String>`	`tokenizeRawTweetText(java.lang.String text)` This is intended for raw tweet text -- we do some HTML entity unescaping before running the tagger.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Class Twokenize