# Uses of Package
opennlp.tools.tokenize

- Packages that use opennlp.tools.tokenize
  Package | Description
  --- | ---
  opennlp.tools.cmdline.parser |
  opennlp.tools.cmdline.tokenizer |
  opennlp.tools.formats | Experimental package related to converting various corpora to OpenNLP Format.
  opennlp.tools.formats.ad | Experimental package related to the `Arvores Deitadas corpus` format.
  opennlp.tools.formats.brat | Experimental package related to the corpus format used by the "brat rapid annotation tool" (brat).
  opennlp.tools.formats.conllu | Experimental package related to the CoNNL-U format.
  opennlp.tools.formats.convert |
  opennlp.tools.formats.irishsentencebank | Experimental package related to the Irish Sentence Bank format.
  opennlp.tools.formats.letsmt | Experimental package related to the `letsmt` corpus format.
  opennlp.tools.formats.masc | Experimental package related to the `MASC` corpus format.
  opennlp.tools.formats.muc | Experimental package related to the `MUC` corpus format.
  opennlp.tools.sentdetect | Package related to identifying sentence boundaries.
  opennlp.tools.tokenize | Contains classes related to finding token or words in a string.
  opennlp.tools.tokenize.lang |
  opennlp.tools.tokenize.lang.en |
  opennlp.tools.util.featuregen | This package contains classes for generating sequence features.

- Classes in opennlp.tools.tokenize used by opennlp.tools.cmdline.parser
  Class | Description
  --- | ---
  Tokenizer | The interface for tokenizers, which segment a string into its tokens.

- Classes in opennlp.tools.tokenize used by opennlp.tools.cmdline.tokenizer
  Class | Description
  --- | ---
  TokenizerEvaluationMonitor | A marker interface for evaluating `tokenizers`.
  TokenizerModel | The `TokenizerModel` is the model used by a learnable `Tokenizer`.
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats
  Class | Description
  --- | ---
  Detokenizer | A `Detokenizer` merges tokens back to their detokenized representation.
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.ad
  Class | Description
  --- | ---
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.brat
  Class | Description
  --- | ---
  Tokenizer | The interface for tokenizers, which segment a string into its tokens.
  TokenizerModel | The `TokenizerModel` is the model used by a learnable `Tokenizer`.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.conllu
  Class | Description
  --- | ---
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.convert
  Class | Description
  --- | ---
  Detokenizer | A `Detokenizer` merges tokens back to their detokenized representation.
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.irishsentencebank
  Class | Description
  --- | ---
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.letsmt
  Class | Description
  --- | ---
  Detokenizer | A `Detokenizer` merges tokens back to their detokenized representation.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.masc
  Class | Description
  --- | ---
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.muc
  Class | Description
  --- | ---
  Tokenizer | The interface for tokenizers, which segment a string into its tokens.

- Classes in opennlp.tools.tokenize used by opennlp.tools.sentdetect
  Class | Description
  --- | ---
  Detokenizer | A `Detokenizer` merges tokens back to their detokenized representation.

- Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize
  Class | Description
  --- | ---
  DetokenizationDictionary |
  DetokenizationDictionary.Operation |
  Detokenizer | A `Detokenizer` merges tokens back to their detokenized representation.
  Detokenizer.DetokenizationOperation | This enum contains an operation for every token to merge the tokens together to their detokenized form.
  SimpleTokenizer | A basic `Tokenizer` implementation which performs tokenization using character classes.
  TokenContextGenerator | Interface for context generators required for `TokenizerME`.
  Tokenizer | The interface for tokenizers, which segment a string into its tokens.
  TokenizerEvaluationMonitor | A marker interface for evaluating `tokenizers`.
  TokenizerFactory | The factory that provides `Tokenizer` default implementation and resources.
  TokenizerModel | The `TokenizerModel` is the model used by a learnable `Tokenizer`.
  TokenSample | A `TokenSample` is text with token spans.
  WhitespaceTokenizer | A basic `Tokenizer` implementation which performs tokenization using white spaces.

- Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize.lang
  Class | Description
  --- | ---
  TokenContextGenerator | Interface for context generators required for `TokenizerME`.

- Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize.lang.en
  Class | Description
  --- | ---
  TokenSample | A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.util.featuregen
  Class | Description
  --- | ---
  Tokenizer | The interface for tokenizers, which segment a string into its tokens.