public class HDFSOffsetWriter extends AbstractOffsetWriter
Modifier and Type | Field and Description |
---|---|
static String |
FILE_FORMAT_EXTENSION
File extension for storing the offsets.
|
static org.joda.time.format.DateTimeFormatter |
FILE_FORMATTER
Formatter to use when creating the file names in a URI compliant format.
|
static String |
PERSIST_TIME_FORMAT
Custom formatter for translating the times into valid file names.
|
Constructor and Description |
---|
HDFSOffsetWriter(org.apache.hadoop.conf.Configuration config,
org.apache.hadoop.fs.Path baseStoragePath)
Creates a writer instance for interacting with the storage specified by the
config and with
the base storage path of baseStoragePath . |
Modifier and Type | Method and Description |
---|---|
void |
close() |
static long |
fileNameToPersistenceTime(String fileName)
Converts a
fileName into the time the offsets were persisted. |
static org.apache.hadoop.fs.Path |
getPersistedTimeStoragePath(org.apache.hadoop.fs.Path baseStoragePath,
long persistedTime)
Creates a
Path for storing the offsets for a specified persistedTime . |
static String |
persistenceTimeToFileName(long persistedTime)
Converts a
persistedTime into a file name for persisting the offsets. |
void |
write(long asOfTime,
Map<org.apache.kafka.common.TopicPartition,Long> offsets)
Persists the
offsets to a configured location with metadata of asOfTime indicating
the time in milliseconds when the offsets were meaningful. |
write
public static final String PERSIST_TIME_FORMAT
public static final org.joda.time.format.DateTimeFormatter FILE_FORMATTER
public static final String FILE_FORMAT_EXTENSION
public HDFSOffsetWriter(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path baseStoragePath)
config
and with
the base storage path of baseStoragePath
.config
- the config for interacting with the underlying data store.baseStoragePath
- the base storage path for offset information.IllegalArgumentException
- if either argument is null
.public void write(long asOfTime, Map<org.apache.kafka.common.TopicPartition,Long> offsets) throws IOException
OffsetWriter
offsets
to a configured location with metadata of asOfTime
indicating
the time in milliseconds when the offsets were meaningful.asOfTime
- the metadata describing when the offsets are accurate as of a time given in milliseconds
since epoch.offsets
- the offsets to persistIOException
- if there is an error persisting the offsets.public void close() throws IOException
IOException
public static org.apache.hadoop.fs.Path getPersistedTimeStoragePath(org.apache.hadoop.fs.Path baseStoragePath, long persistedTime)
Path
for storing the offsets for a specified persistedTime
.baseStoragePath
- The base path the offsets will be stored at.persistedTime
- the time of the data being persisted.IllegalArgumentException
- if the baseStoragePath
is null
.public static long fileNameToPersistenceTime(String fileName)
fileName
into the time the offsets were persisted.fileName
- the file name to parse.IllegalArgumentException
- if the fileName
is not of the correct format or is null
or
empty.public static String persistenceTimeToFileName(long persistedTime)
persistedTime
into a file name for persisting the offsets.persistedTime
- the persisted time to use to generate the file name.Copyright © 2017 The Apache Software Foundation. All rights reserved.