Package opennlp.tools.tokenize

## Class WordpieceTokenizer

- java.lang.Object
- - opennlp.tools.tokenize.WordpieceTokenizer

- All Implemented Interfaces:
    `Tokenizer`

      public class WordpieceTokenizer
      extends Object
      implements Tokenizer

  A `Tokenizer` implementation which performs tokenization using word pieces.

  Adapted under MIT license from https://github.com/robrua/easy-bert.

  For reference see:

  - https://www.tensorflow.org/text/guide/subwords_tokenizer#applying_wordpiece
  - https://cran.r-project.org/web/packages/wordpiece/vignettes/basic_usage.html

- - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `WordpieceTokenizer​(Set<String> vocabulary)` | Initializes a `WordpieceTokenizer` with a `vocabulary` and a default `maxTokenLength` of `50`.
    `WordpieceTokenizer​(Set<String> vocabulary, int maxTokenLength)` | Initializes a `WordpieceTokenizer` with a `vocabulary` and a custom `maxTokenLength`.

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `int` | `getMaxTokenLength()` |
    `String[]` | `tokenize​(String text)` | Splits a string into its atomic parts.
    `Span[]` | `tokenizePos​(String text)` | Finds the boundaries of atomic parts in a string.

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### WordpieceTokenizer

          public WordpieceTokenizer​(Set<String> vocabulary)

      Initializes a `WordpieceTokenizer` with a `vocabulary` and a default `maxTokenLength` of `50`.
      Parameters:
        `vocabulary` - A set of tokens considered the vocabulary.

    - #### WordpieceTokenizer

          public WordpieceTokenizer​(Set<String> vocabulary,
                                    int maxTokenLength)

      Initializes a `WordpieceTokenizer` with a `vocabulary` and a custom `maxTokenLength`.
      Parameters:
        `vocabulary` - A set of tokens considered the vocabulary.
        `maxTokenLength` - A non-negative number that is used as maximum token length.

  - ### Method Detail

    - #### tokenizePos

          public Span[] tokenizePos​(String text)

      Description copied from interface: `Tokenizer`
      Finds the boundaries of atomic parts in a string.
      Specified by:
        `tokenizePos` in interface `Tokenizer`
      Parameters:
        `text` - The string to be tokenized.
      Returns:
        The `spans (offsets into {@code s})` for each token as the individuals array elements.

    - #### tokenize

          public String[] tokenize​(String text)

      Description copied from interface: `Tokenizer`
      Splits a string into its atomic parts.
      Specified by:
        `tokenize` in interface `Tokenizer`
      Parameters:
        `text` - The string to be tokenized.
      Returns:
        The String[] with the individual tokens as the array elements.

    - #### getMaxTokenLength

          public int getMaxTokenLength()

      Returns:
        The maximum token length.