org.apache.crunch.contrib.text

Class Tokenizer

java.lang.Object

org.apache.crunch.contrib.text.Tokenizer

public class Tokenizer
extends Object

Manages a Scanner instance and provides support for returning only a subset of the fields returned by the underlying Scanner.

Constructor Summary

Constructors

Constructor and Description
Tokenizer(Scanner scanner, Set<Integer> indices, boolean keep) Create a new Tokenizer instance.

Method Summary

Modifier and Type	Method and Description
boolean	hasNext() Returns true if the underlying Scanner has any tokens remaining.
String	next() Advance this Tokenizer and return the next String from the Scanner.
Boolean	nextBoolean() Advance this Tokenizer and return the next Boolean from the Scanner.
Double	nextDouble() Advance this Tokenizer and return the next Double from the Scanner.
Float	nextFloat() Advance this Tokenizer and return the next Float from the Scanner.
Integer	nextInt() Advance this Tokenizer and return the next Integer from the Scanner.
Long	nextLong() Advance this Tokenizer and return the next Long from the Scanner.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer(Scanner scanner,
                 Set<Integer> indices,
                 boolean keep)

Create a new Tokenizer instance.

Parameters:

scanner - The scanner to manage

indices - The indices to keep/drop

keep - Whether the indices should be kept (true) or dropped (false)

Method Detail

hasNext

public boolean hasNext()

Returns true if the underlying Scanner has any tokens remaining.

next

public String next()

Advance this Tokenizer and return the next String from the Scanner.

Returns:

The next String from the Scanner

nextLong

public Long nextLong()

Advance this Tokenizer and return the next Long from the Scanner.

Returns:

The next Long from the Scanner

nextBoolean

public Boolean nextBoolean()

Advance this Tokenizer and return the next Boolean from the Scanner.

Returns:

The next Boolean from the Scanner

nextDouble

public Double nextDouble()

Advance this Tokenizer and return the next Double from the Scanner.

Returns:

The next Double from the Scanner

nextFloat

public Float nextFloat()

Advance this Tokenizer and return the next Float from the Scanner.

Returns:

The next Float from the Scanner

nextInt

public Integer nextInt()

Advance this Tokenizer and return the next Integer from the Scanner.

Returns:

The next Integer from the Scanner