org.apache.crunch.types.writable

Class WritableType<T,W extends org.apache.hadoop.io.Writable>

java.lang.Object

org.apache.crunch.types.writable.WritableType<T,W>

All Implemented Interfaces:

Serializable, PType<T>

public class WritableType<T,W extends org.apache.hadoop.io.Writable>
extends Object
implements PType<T>

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
WritableType(Class<T> typeClass, Class<W> writableClass, MapFn<W,T> inputDoFn, MapFn<T,W> outputDoFn, PType... subTypes)

Method Summary

Modifier and Type	Method and Description
ReadableSource<T>	createSourceTarget(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path, Iterable<T> contents, int parallelism) Returns a ReadableSource that contains the data in the given Iterable.
boolean	equals(Object obj)
Converter	getConverter()
ReadableSourceTarget<T>	getDefaultFileSource(org.apache.hadoop.fs.Path path) Returns a SourceTarget that is able to read/write data using the serialization format specified by this PType.
T	getDetachedValue(T value) Returns a copy of a value (or the value itself) that can safely be retained.
PTypeFamily	getFamily() Returns the PTypeFamily that this PType belongs to.
MapFn	getInputMapFn()
MapFn	getOutputMapFn()
Class<W>	getSerializationClass()
List<PType>	getSubTypes() Returns the sub-types that make up this PType if it is a composite instance, such as a tuple.
Class<T>	getTypeClass() Returns the Java type represented by this PType.
int	hashCode()
static <T,W extends org.apache.hadoop.io.Writable> WritableType<T,W>	immutableType(Class<T> typeClass, Class<W> writableClass, MapFn<W,T> inputDoFn, MapFn<T,W> outputDoFn, PType... subTypes) Factory method for a new WritableType instance whose type class is immutable.
void	initialize(org.apache.hadoop.conf.Configuration conf) Initialize this PType for use within a DoFn.

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WritableType

public WritableType(Class<T> typeClass,
                    Class<W> writableClass,
                    MapFn<W,T> inputDoFn,
                    MapFn<T,W> outputDoFn,
                    PType... subTypes)

Method Detail

immutableType

public static <T,W extends org.apache.hadoop.io.Writable> WritableType<T,W> immutableType(Class<T> typeClass,
                                                                                          Class<W> writableClass,
                                                                                          MapFn<W,T> inputDoFn,
                                                                                          MapFn<T,W> outputDoFn,
                                                                                          PType... subTypes)

Factory method for a new WritableType instance whose type class is immutable.

No checking is done to ensure that instances of the type class are immutable, but deep copying will be skipped for instances denoted by the created PType.

getFamily

public PTypeFamily getFamily()

Description copied from interface: PType

Returns the PTypeFamily that this PType belongs to.

Specified by:

getFamily in interface PType<T>

getTypeClass

public Class<T> getTypeClass()

Description copied from interface: PType

Returns the Java type represented by this PType.

Specified by:

getTypeClass in interface PType<T>

getConverter

public Converter getConverter()

Specified by:

getConverter in interface PType<T>

getInputMapFn

public MapFn getInputMapFn()

Specified by:

getInputMapFn in interface PType<T>

getOutputMapFn

public MapFn getOutputMapFn()

Specified by:

getOutputMapFn in interface PType<T>

getSubTypes

public List<PType> getSubTypes()

Description copied from interface: PType

Returns the sub-types that make up this PType if it is a composite instance, such as a tuple.

Specified by:

getSubTypes in interface PType<T>

getSerializationClass

public Class<W> getSerializationClass()

getDefaultFileSource

public ReadableSourceTarget<T> getDefaultFileSource(org.apache.hadoop.fs.Path path)

Description copied from interface: PType

Returns a SourceTarget that is able to read/write data using the serialization format specified by this PType.

Specified by:

getDefaultFileSource in interface PType<T>

createSourceTarget

public ReadableSource<T> createSourceTarget(org.apache.hadoop.conf.Configuration conf,
                                            org.apache.hadoop.fs.Path path,
                                            Iterable<T> contents,
                                            int parallelism)
                                     throws IOException

Description copied from interface: PType

Returns a ReadableSource that contains the data in the given Iterable.

Specified by:

createSourceTarget in interface PType<T>

Parameters:

conf - The Configuration to use

path - The path to write the data to

contents - The contents to write

parallelism - The desired parallelism

Returns:

A new instance of ReadableSource

Throws:

IOException

equals

public boolean equals(Object obj)

Overrides:

equals in class Object

initialize

public void initialize(org.apache.hadoop.conf.Configuration conf)

Description copied from interface: PType

Initialize this PType for use within a DoFn. This generally only needs to be called when using a PType for PType.getDetachedValue(Object).

Specified by:

initialize in interface PType<T>

Parameters:

conf - Configuration object

See Also:

PType.getDetachedValue(Object)

getDetachedValue

public T getDetachedValue(T value)

Description copied from interface: PType

Returns a copy of a value (or the value itself) that can safely be retained.

This is useful when iterable values being processed in a DoFn (via a reducer) need to be held on to for more than the scope of a single iteration, as a reducer (and therefore also a DoFn that has an Iterable as input) re-use deserialized values. More information on object reuse is available in the DoFn class documentation.

Specified by:

getDetachedValue in interface PType<T>

Parameters:

value - The value to be deep-copied

Returns:

A deep copy of the input value

hashCode

public int hashCode()

Overrides:

hashCode in class Object