Package opennlp.tools.tokenize

## Class TokenSample

- java.lang.Object
- - opennlp.tools.tokenize.TokenSample

- All Implemented Interfaces:
    `Serializable`, `Sample`

      public class TokenSample
      extends Object
      implements Sample

  A `TokenSample` is text with token spans.
  See Also:
    Serialized Form

- - ### Field Summary

    Fields
    Modifier and Type | Field | Description
    --- | --- | ---
    `static String` | `DEFAULT_SEPARATOR_CHARS` |

  - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `TokenSample​(String text, Span[] tokenSpans)` | Initializes a `instance`.
    `TokenSample​(Detokenizer detokenizer, String[] tokens)` | Initializes a `instance` via a `Detokenizer`.

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `boolean` | `equals​(Object obj)` |
    `String` | `getText()` |
    `Span[]` | `getTokenSpans()` |
    `int` | `hashCode()` |
    `static TokenSample` | `parse​(String sampleString, String separatorChars)` | Parses a string sample.
    `String` | `toString()` |

    - ### Methods inherited from class java.lang.Object

      `getClass, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### DEFAULT_SEPARATOR_CHARS

          public static final String DEFAULT_SEPARATOR_CHARS

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### TokenSample

          public TokenSample​(String text,
                             Span[] tokenSpans)

      Initializes a `instance`.
      Parameters:
        `text` - The text which contains the tokens. Must not be `null`.
        `tokenSpans` - The spans which mark the start and end of the tokens. Must not be `null`.

    - #### TokenSample

          public TokenSample​(Detokenizer detokenizer,
                             String[] tokens)

      Initializes a `instance` via a `Detokenizer`.
      Parameters:
        `detokenizer` - The text which contains the tokens. Must not be `null`.
        `tokens` - The tokens to be processed. Must not be `null`.

  - ### Method Detail

    - #### getText

          public String getText()

      Returns:
        Retrieves the text.

    - #### getTokenSpans

          public Span[] getTokenSpans()

      Returns:
        Retrieves the token `spans`.

    - #### toString

          public String toString()

      Overrides:
        `toString` in class `Object`

    - #### parse

          public static TokenSample parse​(String sampleString,
                                          String separatorChars)

      Parses a string sample.
      Parameters:
        `sampleString` - The sample to be parsed. Must not be `null`.
        `separatorChars` - The characters to be considered separators. See `DEFAULT_SEPARATOR_CHARS`. Must not be `null`.
      Returns:
        A valid `TokenSample` instance.

    - #### hashCode

          public int hashCode()

      Overrides:
        `hashCode` in class `Object`

    - #### equals

          public boolean equals​(Object obj)

      Overrides:
        `equals` in class `Object`