Package opennlp.tools.tokenize

## Class TokenSampleStream

- java.lang.Object
- - opennlp.tools.util.FilterObjectStream<String,​TokenSample>
  - - opennlp.tools.tokenize.TokenSampleStream

- All Implemented Interfaces:
    `AutoCloseable`, `ObjectStream<TokenSample>`

      public class TokenSampleStream
      extends FilterObjectStream<String,​TokenSample>

  This class is a `stream filter` which reads in string encoded samples and creates `samples` out of them. The input string sample is tokenized if a whitespace or the special separator chars occur.

  Sample:
  "token1 token2 token3<SPLIT>token4"
  The tokens `token1` and `token2` are separated by a whitespace, `token3` and `token4` are separated by the special character sequence. In this case, the default split sequence applies.

  Note: The sequence must be unique in the input string and is not escaped.

- - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `TokenSampleStream​(ObjectStream<String> sentences)` | Initializes a `instance`.
    `TokenSampleStream​(ObjectStream<String> samples, String separatorChars)` | Initializes a `instance`.

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `TokenSample` | `read()` | Returns the next `ObjectStream` object.

    - ### Methods inherited from class opennlp.tools.util.FilterObjectStream

      `close, reset`

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### TokenSampleStream

          public TokenSampleStream​(ObjectStream<String> samples,
                                   String separatorChars)

      Initializes a `instance`.
      Parameters:
        `samples` - A plain text `line stream`. Must not be `null`.
        `separatorChars` - The characters to be considered separators. See `TokenSample.DEFAULT_SEPARATOR_CHARS`. Must not be `null`.

    - #### TokenSampleStream

          public TokenSampleStream​(ObjectStream<String> sentences)

      Initializes a `instance`.
      Parameters:
        `sentences` - A plain text `line stream`. Must not be `null`.

  - ### Method Detail

    - #### read

          public TokenSample read()
                           throws IOException

      Description copied from interface: `ObjectStream`
      Returns the next `ObjectStream` object. Calling this method repeatedly until it returns `null` will return each object from the underlying source exactly once.
      Returns:
        The next object or `null` to signal that the stream is exhausted.
      Throws:
        `IOException` - Thrown if there is an error during reading.