## Uses of Interface
opennlp.tools.tokenize.Tokenizer

- Packages that use Tokenizer
  Package | Description
  --- | ---
  opennlp.tools.cmdline.parser |
  opennlp.tools.formats.brat | Experimental package related to the corpus format used by the "brat rapid annotation tool" (brat).
  opennlp.tools.formats.muc | Experimental package related to the `MUC` corpus format.
  opennlp.tools.tokenize | Contains classes related to finding token or words in a string.
  opennlp.tools.util.featuregen | This package contains classes for generating sequence features.

- - ### Uses of Tokenizer in opennlp.tools.cmdline.parser

    Methods in opennlp.tools.cmdline.parser with parameters of type Tokenizer
    Modifier and Type | Method | Description
    --- | --- | ---
    `static Parse[]` | ParserTool.`parseLine​(String line, Parser parser, Tokenizer tokenizer, int numParses)` |

  - ### Uses of Tokenizer in opennlp.tools.formats.brat

    Constructors in opennlp.tools.formats.brat with parameters of type Tokenizer
    Constructor | Description
    --- | ---
    `BratDocumentParser​(SentenceDetector sentenceDetector, Tokenizer tokenizer)` |
    `BratDocumentParser​(SentenceDetector sentenceDetector, Tokenizer tokenizer, Set<String> nameTypes)` |
    `BratNameSampleStream​(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples)` | Creates a new `BratNameSampleStream`.
    `BratNameSampleStream​(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples, Set<String> nameTypes)` | Creates a new `BratNameSampleStream`.

  - ### Uses of Tokenizer in opennlp.tools.formats.muc

    Constructors in opennlp.tools.formats.muc with parameters of type Tokenizer
    Constructor | Description
    --- | ---
    `MucNameContentHandler​(Tokenizer tokenizer, List<NameSample> storedSamples)` | Initializes a `MucNameContentHandler`.
    `MucNameSampleStream​(Tokenizer tokenizer, ObjectStream<String> samples)` | Initializes a `MucNameSampleStream`.

  - ### Uses of Tokenizer in opennlp.tools.tokenize

    Classes in opennlp.tools.tokenize that implement Tokenizer
    Modifier and Type | Class | Description
    --- | --- | ---
    `class` | `SimpleTokenizer` | A basic `Tokenizer` implementation which performs tokenization using character classes.
    `class` | `TokenizerME` | A `Tokenizer` for converting raw text into separated tokens.
    `class` | `WhitespaceTokenizer` | A basic `Tokenizer` implementation which performs tokenization using white spaces.
    `class` | `WordpieceTokenizer` | A `Tokenizer` implementation which performs tokenization using word pieces.

    Constructors in opennlp.tools.tokenize with parameters of type Tokenizer
    Constructor | Description
    --- | ---
    `TokenizerEvaluator​(Tokenizer tokenizer, TokenizerEvaluationMonitor... listeners)` | Initializes an instance to evaluate a `Tokenizer`.
    `TokenizerStream​(Tokenizer tokenizer, ObjectStream<String> input)` | Initializes a `instance`.

  - ### Uses of Tokenizer in opennlp.tools.util.featuregen

    Constructors in opennlp.tools.util.featuregen with parameters of type Tokenizer
    Constructor | Description
    --- | ---
    `TokenPatternFeatureGenerator​(Tokenizer supportTokenizer)` | Initializes a `TokenPatternFeatureGenerator` instance.