org.apache.crunch.lib

Class DoFns

java.lang.Object

org.apache.crunch.lib.DoFns

public class DoFns
extends Object

Constructor Summary

Constructors

Constructor and Description
DoFns()

Method Summary

Methods

Modifier and Type	Method and Description
static <K,V,T> DoFn<Pair<K,Iterable<V>>,T>	detach(DoFn<Pair<K,Iterable<V>>,T> reduceFn, PType<V> valueType) "Reduce" DoFn wrapper which detaches the values in the iterable, preventing the unexpected behaviour related to object reuse often observed when using Avro.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DoFns

public DoFns()

Method Detail

detach

public static <K,V,T> DoFn<Pair<K,Iterable<V>>,T> detach(DoFn<Pair<K,Iterable<V>>,T> reduceFn,
                                         PType<V> valueType)

"Reduce" DoFn wrapper which detaches the values in the iterable, preventing the unexpected behaviour related to object reuse often observed when using Avro. Wrap your DoFn in a detach(...) and pass in a PType for the Iterable value, and then you'll be handed an Iterable of real distinct objects, instead of the same object being handed to you multiple times with different data. You should use this when you have a parallelDo after a groupBy, and you'd like to capture the objects arriving in the Iterable part of the incoming Pair and pass it through to the output (for example if you want to create an array of outputs from the values to be output as one record). The will incur a performance hit, as it means that every object read from the Iterable will allocate a new Java object for the record and objects for all its non-primitive fields too. If you are rolling up records into a collection then this will be necessary anyway, but if you are only outputting derived data this may impact the speed and memory usage of your job unnecessarily.

Type Parameters:

K - Reduce key

V - Iterable value

T - Output type of DoFn

Parameters:

reduceFn - Underlying DoFn to wrap

valueType - PType of the object contained within the Iterable

Returns:

DoFn which will detach values for you