org.apache.crunch.kafka.offset

Class AbstractOffsetReader

java.lang.Object

org.apache.crunch.kafka.offset.AbstractOffsetReader

All Implemented Interfaces:

Closeable, AutoCloseable, OffsetReader

Direct Known Subclasses:

HDFSOffsetReader

public abstract class AbstractOffsetReader
extends Object
implements OffsetReader

Base implementation of OffsetReader

Constructor Summary

Constructors

Constructor and Description
AbstractOffsetReader()

Method Summary

Modifier and Type	Method and Description
List<Long>	getStoredOffsetPersistenceTimes() Returns the list of available persistence times offsets have been written to the underlying storage mechanism.
Map<org.apache.kafka.common.TopicPartition,Long>	readOffsets(long persistedOffsetTime) Reads the offsets for a given persistedOffsetTime.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.crunch.kafka.offset.OffsetReader

readLatestOffsets

Methods inherited from interface java.io.Closeable

close

Constructor Detail

AbstractOffsetReader

public AbstractOffsetReader()

Method Detail

readOffsets

public Map<org.apache.kafka.common.TopicPartition,Long> readOffsets(long persistedOffsetTime)
                                                             throws IOException

Description copied from interface: OffsetReader

Reads the offsets for a given persistedOffsetTime. Note that not all storage mechanisms support complete historical offset information. Use the OffsetReader.getStoredOffsetPersistenceTimes() to find valid values to specify for persistedOffsetTime.

Specified by:

readOffsets in interface OffsetReader

Parameters:

persistedOffsetTime - the persistence time when offsets were written to the underlying storage system.

Returns:

returns the offsets persisted at the specified persistedOffsetTime. If no offsets were persisted at that time or available to be retrieved then null will be returned.

Throws:

IOException - if there is an error reading from the underlying storage.

getStoredOffsetPersistenceTimes

public List<Long> getStoredOffsetPersistenceTimes()
                                           throws IOException

Description copied from interface: OffsetReader

Returns the list of available persistence times offsets have been written to the underlying storage mechanism. The list of available persistence times will be returned in the order of earliest to latest.

Specified by:

getStoredOffsetPersistenceTimes in interface OffsetReader

Returns:

the collection of persistence times in the form of milliseconds since epoch. If there are no historical persistence times then an empty list is returned.

Throws:

IOException - if there is an error reading from the underlying storage.