Name

adams.flow.transformer.Tokenize

Synopsis

Splits document strings into sentences using the specified document to sentences tokenizer.

Additional information

Flow input/output:
- input: java.lang.String
- output: java.lang.String

Options

• loggingLevel
  

  The logging level for outputting errors and debugging output.

  command-line	-logging-level <OFF|SEVERE|WARNING|INFO|CONFIG|FINE|FINER|FINEST>
  default	WARNING
  min-user-mode	Expert

  
  
• name
  

  The name of the actor.

  command-line	-name <java.lang.String>
  default	Tokenize

  
  
• annotations
  

  The annotations to attach to this actor.

  command-line	-annotation <adams.core.base.BaseAnnotation>
  default

  
  
• skip
  

  If set to true, transformation is skipped and the input token is just forwarded as it is.

  command-line	-skip <boolean>
  default	false

  
  
• stopFlowOnError
  

  If set to true, the flow execution at this level gets stopped in case this actor encounters an error; the error gets propagated; useful for critical actors.

  command-line	-stop-flow-on-error <boolean>
  default	false
  min-user-mode	Expert

  
  
• silent
  

  If enabled, then no errors are output in the console; Note: the enclosing actor handler must have this enabled as well.

  command-line	-silent <boolean>
  default	false
  min-user-mode	Expert

  
  
• outputArray
  

  Whether to output the sentences as array or one-by-one.

  command-line	-output-array <boolean>
  default	false

  
  
• tokenizer
  

  The tokenizer to use for generating words from strings.

  command-line	-tokenizer <adams.flow.transformer.tokenizer.AbstractTokenizer>
  default	adams.flow.transformer.tokenizer.StanfordPTBTokenizer