org.apache.crunch.kafka.offset.hdfs

Class HDFSOffsetReader

java.lang.Object

org.apache.crunch.kafka.offset.AbstractOffsetReader

org.apache.crunch.kafka.offset.hdfs.HDFSOffsetReader

All Implemented Interfaces:

Closeable, AutoCloseable, OffsetReader

public class HDFSOffsetReader
extends AbstractOffsetReader

Reader implementation that reads offset information from HDFS.

Constructor Summary

Constructors

Constructor and Description
HDFSOffsetReader(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path baseOffsetStoragePath) Creates a reader instance for interacting with the storage specified by the config and with the base storage path of baseStoragePath.

Method Summary

Modifier and Type	Method and Description
void	close()
List<Long>	getStoredOffsetPersistenceTimes() Returns the list of available persistence times offsets have been written to the underlying storage mechanism.
Map<org.apache.kafka.common.TopicPartition,Long>	readLatestOffsets() Reads the last stored offsets.
Map<org.apache.kafka.common.TopicPartition,Long>	readOffsets(long persistedOffsetTime) Reads the offsets for a given persistedOffsetTime.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HDFSOffsetReader

public HDFSOffsetReader(org.apache.hadoop.conf.Configuration config,
                        org.apache.hadoop.fs.Path baseOffsetStoragePath)

Creates a reader instance for interacting with the storage specified by the config and with the base storage path of baseStoragePath.

Parameters:

config - the config for interacting with the underlying data store.

baseOffsetStoragePath - the base storage path for offset information. If the path does not exist it will be created.

Throws:

IllegalArgumentException - if either argument is null.

Method Detail

readLatestOffsets

public Map<org.apache.kafka.common.TopicPartition,Long> readLatestOffsets()
                                                                   throws IOException

Description copied from interface: OffsetReader

Reads the last stored offsets.

Returns:

the last stored offsets. If there are no stored offsets an empty collection will be returned.

Throws:

IOException - if there is an error reading from the underlying storage.

readOffsets

public Map<org.apache.kafka.common.TopicPartition,Long> readOffsets(long persistedOffsetTime)
                                                             throws IOException

Description copied from interface: OffsetReader

Reads the offsets for a given persistedOffsetTime. Note that not all storage mechanisms support complete historical offset information. Use the OffsetReader.getStoredOffsetPersistenceTimes() to find valid values to specify for persistedOffsetTime.

Specified by:

readOffsets in interface OffsetReader

Overrides:

readOffsets in class AbstractOffsetReader

Parameters:

persistedOffsetTime - the persistence time when offsets were written to the underlying storage system.

Returns:

returns the offsets persisted at the specified persistedOffsetTime. If no offsets were persisted at that time or available to be retrieved then null will be returned.

Throws:

IOException - if there is an error reading from the underlying storage.

getStoredOffsetPersistenceTimes

public List<Long> getStoredOffsetPersistenceTimes()
                                           throws IOException

Description copied from interface: OffsetReader

Returns the list of available persistence times offsets have been written to the underlying storage mechanism. The list of available persistence times will be returned in the order of earliest to latest.

Specified by:

getStoredOffsetPersistenceTimes in interface OffsetReader

Overrides:

getStoredOffsetPersistenceTimes in class AbstractOffsetReader

Returns:

the collection of persistence times in the form of milliseconds since epoch. If there are no historical persistence times then an empty list is returned.

Throws:

IOException - if there is an error reading from the underlying storage.

close

public void close()
           throws IOException

Throws:

IOException