# Uses of Package
opennlp.tools.tokenize

- Packages that use opennlp.tools.tokenize
  Package | Description
  --- | ---
  opennlp.tools.cmdline.parser |
  opennlp.tools.cmdline.tokenizer |
  opennlp.tools.formats | Experimental package related to converting various corpora to OpenNLP Format.
  opennlp.tools.formats.ad |
  opennlp.tools.formats.brat |
  opennlp.tools.formats.conllu |
  opennlp.tools.formats.convert |
  opennlp.tools.formats.irishsentencebank |
  opennlp.tools.formats.letsmt |
  opennlp.tools.formats.masc |
  opennlp.tools.formats.muc |
  opennlp.tools.sentdetect | Package related to identifying sentece boundries.
  opennlp.tools.tokenize | Contains classes related to finding token or words in a string.
  opennlp.tools.tokenize.lang |
  opennlp.tools.tokenize.lang.en |
  opennlp.tools.util.featuregen | This package contains classes for generating sequence features.

- Classes in opennlp.tools.tokenize used by opennlp.tools.cmdline.parser
  Class | Description
  --- | ---
  Tokenizer | The interface for tokenizers, which segment a string into its tokens.

- Classes in opennlp.tools.tokenize used by opennlp.tools.cmdline.tokenizer
  Class | Description
  --- | ---
  TokenizerEvaluationMonitor |
  TokenizerModel | The `TokenizerModel` is the model used by a learnable `Tokenizer`.
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats
  Class | Description
  --- | ---
  Detokenizer | A Detokenizer merges tokens back to their untokenized representation.
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.ad
  Class | Description
  --- | ---
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.brat
  Class | Description
  --- | ---
  Tokenizer | The interface for tokenizers, which segment a string into its tokens.
  TokenizerModel | The `TokenizerModel` is the model used by a learnable `Tokenizer`.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.conllu
  Class | Description
  --- | ---
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.convert
  Class | Description
  --- | ---
  Detokenizer | A Detokenizer merges tokens back to their untokenized representation.
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.irishsentencebank
  Class | Description
  --- | ---
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.letsmt
  Class | Description
  --- | ---
  Detokenizer | A Detokenizer merges tokens back to their untokenized representation.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.masc
  Class | Description
  --- | ---
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.muc
  Class | Description
  --- | ---
  Tokenizer | The interface for tokenizers, which segment a string into its tokens.

- Classes in opennlp.tools.tokenize used by opennlp.tools.sentdetect
  Class | Description
  --- | ---
  Detokenizer | A Detokenizer merges tokens back to their untokenized representation.

- Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize
  Class | Description
  --- | ---
  DetokenizationDictionary |
  DetokenizationDictionary.Operation |
  Detokenizer | A Detokenizer merges tokens back to their untokenized representation.
  Detokenizer.DetokenizationOperation | This enum contains an operation for every token to merge the tokens together to their detokenized form.
  SimpleTokenizer | Performs tokenization using character classes.
  TokenContextGenerator | Interface for `TokenizerME` context generators.
  Tokenizer | The interface for tokenizers, which segment a string into its tokens.
  TokenizerEvaluationMonitor |
  TokenizerFactory | The factory that provides `Tokenizer` default implementations and resources.
  TokenizerModel | The `TokenizerModel` is the model used by a learnable `Tokenizer`.
  TokenSample | A `TokenSample` is text with token spans.
  WhitespaceTokenizer | This tokenizer uses white spaces to tokenize the input text.

- Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize.lang
  Class | Description
  --- | ---
  TokenContextGenerator | Interface for `TokenizerME` context generators.

- Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize.lang.en
  Class | Description
  --- | ---
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.util.featuregen
  Class | Description
  --- | ---
  Tokenizer | The interface for tokenizers, which segment a string into its tokens.