Package opennlp.tools.tokenize

## Interface Detokenizer

- All Known Implementing Classes:
    `DictionaryDetokenizer`

      public interface Detokenizer

  A Detokenizer merges tokens back to their untokenized representation.

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Interface | Description
    --- | --- | ---
    `static class` | `Detokenizer.DetokenizationOperation` | This enum contains an operation for every token to merge the tokens together to their detokenized form.

  - ### Method Summary

    All Methods Instance Methods Abstract Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `Detokenizer.DetokenizationOperation[]` | `detokenize​(String[] tokens)` | Detokenize the input tokens.
    `String` | `detokenize​(String[] tokens, String splitMarker)` | Detokenize the input tokens into a String.

- - ### Method Detail

    - #### detokenize

          Detokenizer.DetokenizationOperation[] detokenize​(String[] tokens)

      Detokenize the input tokens.
      Parameters:
        `tokens` - the tokens to detokenize.
      Returns:
        the merge operations to detokenize the input tokens.

    - #### detokenize

          String detokenize​(String[] tokens,
                            String splitMarker)

      Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.
      Parameters:
        `tokens` - the token which should be concatenated
        `splitMarker` - the split marker or null
      Returns:
        the concatenated tokens