Package opennlp.tools.tokenize

## Interface Detokenizer

- All Known Implementing Classes:
    `DictionaryDetokenizer`

      public interface Detokenizer

  A `Detokenizer` merges tokens back to their detokenized representation.

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Interface | Description
    --- | --- | ---
    `static class` | `Detokenizer.DetokenizationOperation` | This enum contains an operation for every token to merge the tokens together to their detokenized form.

  - ### Method Summary

    All Methods Instance Methods Abstract Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `Detokenizer.DetokenizationOperation[]` | `detokenize​(String[] tokens)` | Detokenizes the collection of tokens.
    `String` | `detokenize​(String[] tokens, String splitMarker)` | Detokenizes the input `tokens` into a String.

- - ### Method Detail

    - #### detokenize

          Detokenizer.DetokenizationOperation[] detokenize​(String[] tokens)

      Detokenizes the collection of tokens.
      Parameters:
        `tokens` - The elements which should be detokenized.
      Returns:
        The `merge operations` to handle given `tokens`.

    - #### detokenize

          String detokenize​(String[] tokens,
                            String splitMarker)

      Detokenizes the input `tokens` into a String. Tokens which are connected without a `whitespace` character in between can be separated by a given `splitMarker`.
      Parameters:
        `tokens` - The elements which should be concatenated.
        `splitMarker` - The split marker or `null`.
      Returns:
        The concatenated tokens as a single string.