org.apache.crunch.kafka.offset

Interface OffsetReader

All Superinterfaces:

AutoCloseable, Closeable

All Known Implementing Classes:

AbstractOffsetReader, HDFSOffsetReader

public interface OffsetReader
extends Closeable

Reader API that supports reading offset information from an underlying storage mechanism.

Method Summary

Modifier and Type	Method and Description
List<Long>	getStoredOffsetPersistenceTimes() Returns the list of available persistence times offsets have been written to the underlying storage mechanism.
Map<org.apache.kafka.common.TopicPartition,Long>	readLatestOffsets() Reads the last stored offsets.
Map<org.apache.kafka.common.TopicPartition,Long>	readOffsets(long persistedOffsetTime) Reads the offsets for a given persistedOffsetTime.

Methods inherited from interface java.io.Closeable

close

Method Detail

readLatestOffsets

Map<org.apache.kafka.common.TopicPartition,Long> readLatestOffsets()
                                                            throws IOException

Reads the last stored offsets.

Returns:

the last stored offsets. If there are no stored offsets an empty collection will be returned.

Throws:

IOException - if there is an error reading from the underlying storage.

readOffsets

Map<org.apache.kafka.common.TopicPartition,Long> readOffsets(long persistedOffsetTime)
                                                      throws IOException

Reads the offsets for a given persistedOffsetTime. Note that not all storage mechanisms support complete historical offset information. Use the getStoredOffsetPersistenceTimes() to find valid values to specify for persistedOffsetTime.

Parameters:

persistedOffsetTime - the persistence time when offsets were written to the underlying storage system.

Returns:

returns the offsets persisted at the specified persistedOffsetTime. If no offsets were persisted at that time or available to be retrieved then null will be returned.

Throws:

IOException - if there is an error reading from the underlying storage.

getStoredOffsetPersistenceTimes

List<Long> getStoredOffsetPersistenceTimes()
                                    throws IOException

Returns the list of available persistence times offsets have been written to the underlying storage mechanism. The list of available persistence times will be returned in the order of earliest to latest.

Returns:

the collection of persistence times in the form of milliseconds since epoch. If there are no historical persistence times then an empty list is returned.

Throws:

IOException - if there is an error reading from the underlying storage.