Package opennlp.tools.tokenize

## Class TokenizerModel

- java.lang.Object
- - opennlp.tools.util.model.BaseModel
  - - opennlp.tools.tokenize.TokenizerModel

- All Implemented Interfaces:
    `Serializable`, `ArtifactProvider`

      public final class TokenizerModel
      extends BaseModel

  The `TokenizerModel` is the model used by a learnable `Tokenizer`.
  See Also:
    `TokenizerME`, Serialized Form

- - ### Field Summary

    - ### Fields inherited from class opennlp.tools.util.model.BaseModel

      `TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY`

  - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `TokenizerModel​(File modelFile)` | Initializes the current instance.
    `TokenizerModel​(InputStream in)` | Initializes the current instance.
    `TokenizerModel​(URL modelURL)` | Initializes the current instance.
    `TokenizerModel​(Path modelPath)` |
    `TokenizerModel​(MaxentModel tokenizerModel, Map<String,​String> manifestInfoEntries, TokenizerFactory tokenizerFactory)` | Initializes the current instance.

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `Dictionary` | `getAbbreviations()` |
    `TokenizerFactory` | `getFactory()` |
    `MaxentModel` | `getMaxentModel()` |
    `boolean` | `useAlphaNumericOptimization()` |

    - ### Methods inherited from class opennlp.tools.util.model.BaseModel

      `getArtifact, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serialize, serialize, serialize`

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### TokenizerModel

          public TokenizerModel​(MaxentModel tokenizerModel,
                                Map<String,​String> manifestInfoEntries,
                                TokenizerFactory tokenizerFactory)

      Initializes the current instance.
      Parameters:
        `tokenizerModel` - the model
        `manifestInfoEntries` - the manifest
        `tokenizerFactory` - the factory

    - #### TokenizerModel

          public TokenizerModel​(InputStream in)
                         throws IOException

      Initializes the current instance.
      Parameters:
        `in` - the Input Stream to load the model from
      Throws:
        `IOException` - if reading from the stream fails in anyway
        `InvalidFormatException` - if the stream doesn't have the expected format

    - #### TokenizerModel

          public TokenizerModel​(File modelFile)
                         throws IOException

      Initializes the current instance.
      Parameters:
        `modelFile` - the file containing the tokenizer model
      Throws:
        `IOException` - if reading from the stream fails in anyway

    - #### TokenizerModel

          public TokenizerModel​(Path modelPath)
                         throws IOException

      Throws:
        `IOException`

    - #### TokenizerModel

          public TokenizerModel​(URL modelURL)
                         throws IOException

      Initializes the current instance.
      Parameters:
        `modelURL` - the URL pointing to the tokenizer model
      Throws:
        `IOException` - if reading from the stream fails in anyway

  - ### Method Detail

    - #### getFactory

          public TokenizerFactory getFactory()

    - #### getMaxentModel

          public MaxentModel getMaxentModel()

    - #### getAbbreviations

          public Dictionary getAbbreviations()

    - #### useAlphaNumericOptimization

          public boolean useAlphaNumericOptimization()