Package opennlp.tools.tokenize

## Class TokenizerME

- java.lang.Object
- - opennlp.tools.tokenize.TokenizerME

- All Implemented Interfaces:
    `Tokenizer`

      public class TokenizerME
      extends Object

  A `Tokenizer` for converting raw text into separated tokens. It uses Maximum Entropy to make its decisions. The features are loosely based off of Jeff Reynar's UPenn thesis "Topic Segmentation: Algorithms and Applications.", which is available from his homepage: http://www.cis.upenn.edu/~jcreynar.

  This implementation needs a statistical model to tokenize a text which reproduces the tokenization observed in the training data used to create the model. The `TokenizerModel` class encapsulates that model and provides methods to create it from the binary representation.

  A tokenizer instance is not thread-safe. For each thread, one tokenizer must be instantiated which can share one `TokenizerModel` instance to safe memory.

  To train a new model, the `train(ObjectStream, TokenizerFactory, TrainingParameters)` method can be used.

  Sample usage:

  `InputStream modelIn;
  ...
  TokenizerModel model = TokenizerModel(modelIn);
  Tokenizer tokenizer = new TokenizerME(model);
  String tokens[] = tokenizer.tokenize("A sentence to be tokenized.");`

  See Also:
    `Tokenizer`, `TokenizerModel`, `TokenSample`

- - ### Field Summary

    Fields
    Modifier and Type | Field | Description
    --- | --- | ---
    `static String` | `NO_SPLIT` | Constant indicates no token split.
    `static String` | `SPLIT` | Constant indicates a token split.

  - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `TokenizerME​(String language)` | Initializes a `TokenizerME` by downloading a default model.
    `TokenizerME​(TokenizerModel model)` | Instantiates a `TokenizerME` with an existing `TokenizerModel`.
    `TokenizerME​(TokenizerModel model, Factory factory)` | Deprecated. use `TokenizerFactory` to extend the Tokenizer functionality

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `double[]` | `getTokenProbabilities()` |
    `void` | `setKeepNewLines​(boolean keepNewLines)` | Switches whether to keep new lines or not.
    `String[]` | `tokenize​(String s)` | Splits a string into its atomic parts.
    `Span[]` | `tokenizePos​(String d)` | Tokenizes the string.
    `static TokenizerModel` | `train​(ObjectStream<TokenSample> samples, TokenizerFactory factory, TrainingParameters mlParams)` | Trains a model for the `TokenizerME`.
    `boolean` | `useAlphaNumericOptimization()` |

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### SPLIT

          public static final String SPLIT

      Constant indicates a token split.
      See Also:
        Constant Field Values

    - #### NO_SPLIT

          public static final String NO_SPLIT

      Constant indicates no token split.
      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### TokenizerME

          public TokenizerME​(String language)
                      throws IOException

      Initializes a `TokenizerME` by downloading a default model.
      Parameters:
        `language` - The language of the tokenizer.
      Throws:
        `IOException` - Thrown if the model cannot be downloaded or saved.

    - #### TokenizerME

          public TokenizerME​(TokenizerModel model)

      Instantiates a `TokenizerME` with an existing `TokenizerModel`.
      Parameters:
        `model` - The `TokenizerModel` to be used.

    - #### TokenizerME

          @Deprecated
          public TokenizerME​(TokenizerModel model,
                             Factory factory)

      Deprecated.
      use `TokenizerFactory` to extend the Tokenizer functionality

  - ### Method Detail

    - #### getTokenProbabilities

          public double[] getTokenProbabilities()

      Returns:
        the probabilities associated with the most recent calls to `Tokenizer.tokenize(String)` or `tokenizePos(String)`. If not applicable an empty array is returned.

    - #### tokenizePos

          public Span[] tokenizePos​(String d)

      Tokenizes the string.
      Parameters:
        `d` - The string to be tokenized.
      Returns:
        A `Span` array containing individual tokens as elements.

    - #### train

          public static TokenizerModel train​(ObjectStream<TokenSample> samples,
                                             TokenizerFactory factory,
                                             TrainingParameters mlParams)
                                      throws IOException

      Trains a model for the `TokenizerME`.
      Parameters:
        `samples` - The samples used for the training.
        `factory` - A `TokenizerFactory` to get resources from.
        `mlParams` - The machine learning `train parameters`.
      Returns:
        A trained `TokenizerModel`.
      Throws:
        `IOException` - Thrown during IO operations on a temp file which is created during training. Or if reading from the `ObjectStream` fails.

    - #### useAlphaNumericOptimization

          public boolean useAlphaNumericOptimization()

      Returns:
        `true` if the tokenizer uses alphanumeric optimization, `false` otherwise.

    - #### tokenize

          public String[] tokenize​(String s)

      Description copied from interface: `Tokenizer`
      Splits a string into its atomic parts.
      Specified by:
        `tokenize` in interface `Tokenizer`
      Parameters:
        `s` - The string to be tokenized.
      Returns:
        The String[] with the individual tokens as the array elements.

    - #### setKeepNewLines

          public void setKeepNewLines​(boolean keepNewLines)

      Switches whether to keep new lines or not.
      Parameters:
        `keepNewLines` - `True` if new lines are kept, `false` otherwise.