Class WordTokenSource

java.lang.Object

WordTokenSource

public class WordTokenSource
extends java.lang.Object

Provides a way of reading single tokens from a stream. Has knowledge of a Settings object, providing all tokens and sentence splitting rules.

Constructor Summary

Constructors

Constructor and Description
WordTokenSource(Settings s, java.io.BufferedReader bin) Creates a WordTokenSource with some token-related settings and a source of data.

Method Summary

Methods

Modifier and Type	Method and Description
Token	readToken() Acquires a token from the source.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WordTokenSource

public WordTokenSource(Settings s,
               java.io.BufferedReader bin)

Creates a WordTokenSource with some token-related settings and a source of data.

Parameters:

s - The Settings object that defines EOS split points and other token items.

bin - Where to read data, as a buffered reader.

Method Detail

readToken

public Token readToken()
                throws java.io.IOException

Acquires a token from the source. This handles all reading/writing from the source reader, reading a sentence when required and caching the words.

Returns:

A token from the source, or null if there are none left.

Throws:

java.io.IOException - if the BufferedReader given in the constructor raises some error.