# Uses of Interface
opennlp.tools.tokenize.Tokenizer

Packages that use Tokenizer
Package
Description
opennlp.tools.cmdline.parser
opennlp.tools.formats.brat
Experimental package related to the corpus format used by the "brat rapid annotation tool" (brat).
opennlp.tools.formats.muc
Experimental package related to the `MUC` corpus format.
opennlp.tools.tokenize
Contains classes related to finding token or words in a string.
opennlp.tools.util.featuregen
This package contains classes for generating sequence features.

- ## Uses of Tokenizer in opennlp.tools.cmdline.parser

  Methods in opennlp.tools.cmdline.parser with parameters of type Tokenizer
  Modifier and Type
  Method
  Description
  `static Parse[]`
  ParserTool.`parseLine(String line, Parser parser, Tokenizer tokenizer, int numParses)`

- ## Uses of Tokenizer in opennlp.tools.formats.brat

  Constructors in opennlp.tools.formats.brat with parameters of type Tokenizer
  Modifier
  Constructor
  Description
  ``
  `BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer)`
  ``
  `BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer, Set<String> nameTypes)`
  ``
  `BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples)`
  Creates a new `BratNameSampleStream`.
  ``
  `BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples, Set<String> nameTypes)`
  Creates a new `BratNameSampleStream`.

- ## Uses of Tokenizer in opennlp.tools.formats.muc

  Constructors in opennlp.tools.formats.muc with parameters of type Tokenizer
  Modifier
  Constructor
  Description
  ``
  `MucNameContentHandler(Tokenizer tokenizer, List<NameSample> storedSamples)`
  Initializes a `MucNameContentHandler`.
  `protected`
  `MucNameSampleStream(Tokenizer tokenizer, ObjectStream<String> samples)`
  Initializes a `MucNameSampleStream`.

- ## Uses of Tokenizer in opennlp.tools.tokenize

  Classes in opennlp.tools.tokenize that implement Tokenizer
  Modifier and Type
  Class
  Description
  `class`
  `SimpleTokenizer`
  A basic `Tokenizer` implementation which performs tokenization using character classes.
  `class`
  `TokenizerME`
  A `Tokenizer` for converting raw text into separated tokens.
  `class`
  `WhitespaceTokenizer`
  A basic `Tokenizer` implementation which performs tokenization using white spaces.
  `class`
  `WordpieceTokenizer`
  A `Tokenizer` implementation which performs tokenization using word pieces.
  Constructors in opennlp.tools.tokenize with parameters of type Tokenizer
  Modifier
  Constructor
  Description
  ``
  `TokenizerEvaluator(Tokenizer tokenizer, TokenizerEvaluationMonitor... listeners)`
  Initializes an instance to evaluate a `Tokenizer`.
  ``
  `TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input)`
  Initializes a `instance`.

- ## Uses of Tokenizer in opennlp.tools.util.featuregen

  Constructors in opennlp.tools.util.featuregen with parameters of type Tokenizer
  Modifier
  Constructor
  Description
  ``
  `TokenPatternFeatureGenerator(Tokenizer supportTokenizer)`
  Initializes a `TokenPatternFeatureGenerator` instance.