This project has retired. For details please refer to its Attic page.

org.apache.crunch
Class CombineFn<S,T>

java.lang.Object
  org.apache.crunch.DoFn<Pair<S,Iterable<T>>,Pair<S,T>>
      org.apache.crunch.CombineFn<S,T>

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

Aggregate.TopKCombineFn, CombineFn.AggregatorCombineFn

public abstract class CombineFn<S,T>
extends DoFn<Pair<S,Iterable<T>>,Pair<S,T>>

A special DoFn implementation that converts an Iterable of values into a single value. If a CombineFn instance is used on a PGroupedTable, the function will be applied to the output of the map stage before the data is passed to the reducer, which can improve the runtime of certain classes of jobs.

Note that the incoming Iterable can only be used to create an Iterator once. Calling Iterable.iterator() method a second time will throw an IllegalStateException.

See Also:

Serialized Form

Nested Class Summary

static interface	CombineFn.Aggregator<T> Deprecated. Use Aggregator
static class	CombineFn.AggregatorCombineFn<K,V> Deprecated. Use the Aggregators.toCombineFn(org.apache.crunch.Aggregator) adapter
static interface	CombineFn.AggregatorFactory<T> Deprecated. Use PGroupedTable.combineValues(Aggregator) which doesn't require a factory.
static class	CombineFn.FirstNAggregator<V> Deprecated. Use Aggregators.FIRST_N(int)
static class	CombineFn.LastNAggregator<V> Deprecated. Use Aggregators.LAST_N(int)
static class	CombineFn.MaxBigInts Deprecated. Use Aggregators.MAX_BIGINTS()
static class	CombineFn.MaxDoubles Deprecated. Use Aggregators.MAX_DOUBLES()
static class	CombineFn.MaxFloats Deprecated. Use Aggregators.MAX_FLOATS()
static class	CombineFn.MaxInts Deprecated. Use Aggregators.MAX_INTS()
static class	CombineFn.MaxLongs Deprecated. Use Aggregators.MAX_LONGS()
static class	CombineFn.MaxNAggregator<V extends Comparable<V>> Deprecated. Use Aggregators.MAX_N(int, Class)
static class	CombineFn.MinBigInts Deprecated. Use Aggregators.MIN_BIGINTS()
static class	CombineFn.MinDoubles Deprecated. Use Aggregators.MIN_DOUBLES()
static class	CombineFn.MinFloats Deprecated. Use Aggregators.MIN_FLOATS()
static class	CombineFn.MinInts Deprecated. Use Aggregators.MIN_INTS()
static class	CombineFn.MinLongs Deprecated. Use Aggregators.MIN_LONGS()
static class	CombineFn.MinNAggregator<V extends Comparable<V>> Deprecated. Use Aggregators.MIN_N(int, Class)
static class	CombineFn.PairAggregator<V1,V2> Deprecated. Use Aggregators.pairAggregator(Aggregator, Aggregator)
static class	CombineFn.QuadAggregator<A,B,C,D> Deprecated. Use Aggregators.quadAggregator(Aggregator, Aggregator, Aggregator, Aggregator)
static class	CombineFn.SimpleAggregator<T> Deprecated. Use Aggregators.SimpleAggregator
static class	CombineFn.StringConcatAggregator Deprecated. Use Aggregators.STRING_CONCAT(String, boolean, long, long)
static class	CombineFn.SumBigInts Deprecated. Use Aggregators.SUM_BIGINTS()
static class	CombineFn.SumDoubles Deprecated. Use Aggregators.SUM_DOUBLES()
static class	CombineFn.SumFloats Deprecated. Use Aggregators.SUM_FLOATS()
static class	CombineFn.SumInts Deprecated. Use Aggregators.SUM_INTS()
static class	CombineFn.SumLongs Deprecated. Use Aggregators.SUM_LONGS()
static class	CombineFn.TripAggregator<A,B,C> Deprecated. Use Aggregators.tripAggregator(Aggregator, Aggregator, Aggregator)
static class	CombineFn.TupleNAggregator Deprecated. Use Aggregators.tupleAggregator(Aggregator...)

Field Summary

static CombineFn.AggregatorFactory<BigInteger>	MAX_BIGINTS Deprecated. Use Aggregators.MAX_BIGINTS()
static CombineFn.AggregatorFactory<Double>	MAX_DOUBLES Deprecated. Use Aggregators.MAX_DOUBLES()
static CombineFn.AggregatorFactory<Float>	MAX_FLOATS Deprecated. Use Aggregators.MAX_FLOATS()
static CombineFn.AggregatorFactory<Integer>	MAX_INTS Deprecated. Use Aggregators.MAX_INTS()
static CombineFn.AggregatorFactory<Long>	MAX_LONGS Deprecated. Use Aggregators.MAX_LONGS()
static CombineFn.AggregatorFactory<BigInteger>	MIN_BIGINTS Deprecated. Use Aggregators.MIN_BIGINTS()
static CombineFn.AggregatorFactory<Double>	MIN_DOUBLES Deprecated. Use Aggregators.MIN_DOUBLES()
static CombineFn.AggregatorFactory<Float>	MIN_FLOATS Deprecated. Use Aggregators.MIN_FLOATS()
static CombineFn.AggregatorFactory<Integer>	MIN_INTS Deprecated. Use Aggregators.MIN_INTS()
static CombineFn.AggregatorFactory<Long>	MIN_LONGS Deprecated. Use Aggregators.MIN_LONGS()
static CombineFn.AggregatorFactory<BigInteger>	SUM_BIGINTS Deprecated. Use Aggregators.SUM_BIGINTS()
static CombineFn.AggregatorFactory<Double>	SUM_DOUBLES Deprecated. Use Aggregators.SUM_DOUBLES()
static CombineFn.AggregatorFactory<Float>	SUM_FLOATS Deprecated. Use Aggregators.SUM_FLOATS()
static CombineFn.AggregatorFactory<Integer>	SUM_INTS Deprecated. Use Aggregators.SUM_INTS()
static CombineFn.AggregatorFactory<Long>	SUM_LONGS Deprecated. Use Aggregators.SUM_LONGS()

Constructor Summary

CombineFn()

Method Summary

static <K,V> CombineFn<K,V>	aggregator(CombineFn.Aggregator<V> aggregator) Deprecated. Use Aggregators.toCombineFn(Aggregator)
static <K,V> CombineFn<K,V>	aggregatorFactory(CombineFn.AggregatorFactory<V> aggregator) Deprecated. Use PGroupedTable.combineValues(Aggregator) which doesn't require a factory.
static <K,V> CombineFn<K,V>	FIRST_N(int n) Deprecated. Use Aggregators.FIRST_N(int)
static <K,V> CombineFn<K,V>	LAST_N(int n) Deprecated. Use Aggregators.LAST_N(int)
static <K> CombineFn<K,BigInteger>	MAX_BIGINTS() Deprecated. Use Aggregators.MAX_BIGINTS()
static <K> CombineFn<K,BigInteger>	MAX_BIGINTS(int n) Deprecated. Use Aggregators.MAX_BIGINTS(int)
static <K> CombineFn<K,Double>	MAX_DOUBLES() Deprecated. Use Aggregators.MAX_DOUBLES()
static <K> CombineFn<K,Double>	MAX_DOUBLES(int n) Deprecated. Use Aggregators.MAX_DOUBLES(int)
static <K> CombineFn<K,Float>	MAX_FLOATS() Deprecated. Use Aggregators.MAX_FLOATS()
static <K> CombineFn<K,Float>	MAX_FLOATS(int n) Deprecated. Use Aggregators.MAX_FLOATS(int)
static <K> CombineFn<K,Integer>	MAX_INTS() Deprecated. Use Aggregators.MAX_INTS()
static <K> CombineFn<K,Integer>	MAX_INTS(int n) Deprecated. Use Aggregators.MAX_INTS(int)
static <K> CombineFn<K,Long>	MAX_LONGS() Deprecated. Use Aggregators.MAX_LONGS()
static <K> CombineFn<K,Long>	MAX_LONGS(int n) Deprecated. Use Aggregators.MAX_LONGS(int)
static <K> CombineFn<K,BigInteger>	MIN_BIGINTS() Deprecated. Use Aggregators.MIN_BIGINTS()
static <K> CombineFn<K,BigInteger>	MIN_BIGINTS(int n) Deprecated. Use Aggregators.MIN_BIGINTS(int)
static <K> CombineFn<K,Double>	MIN_DOUBLES() Deprecated. Use Aggregators.MIN_DOUBLES()
static <K> CombineFn<K,Double>	MIN_DOUBLES(int n) Deprecated. Use Aggregators.MIN_DOUBLES(int)
static <K> CombineFn<K,Float>	MIN_FLOATS() Deprecated. Use Aggregators.MIN_FLOATS()
static <K> CombineFn<K,Float>	MIN_FLOATS(int n) Deprecated. Use Aggregators.MIN_FLOATS(int)
static <K> CombineFn<K,Integer>	MIN_INTS() Deprecated. Use Aggregators.MIN_INTS()
static <K> CombineFn<K,Integer>	MIN_INTS(int n) Deprecated. Use Aggregators.MIN_INTS(int)
static <K> CombineFn<K,Long>	MIN_LONGS() Deprecated. Use Aggregators.MIN_LONGS()
static <K> CombineFn<K,Long>	MIN_LONGS(int n) Deprecated. Use Aggregators.MIN_LONGS(int)
static <K,V1,V2> CombineFn<K,Pair<V1,V2>>	pairAggregator(CombineFn.AggregatorFactory<V1> a1, CombineFn.AggregatorFactory<V2> a2) Deprecated. Use Aggregators.pairAggregator(Aggregator, Aggregator)
static <K,A,B,C,D> CombineFn<K,Tuple4<A,B,C,D>>	quadAggregator(CombineFn.AggregatorFactory<A> a1, CombineFn.AggregatorFactory<B> a2, CombineFn.AggregatorFactory<C> a3, CombineFn.AggregatorFactory<D> a4) Deprecated. Use Aggregators.quadAggregator(Aggregator, Aggregator, Aggregator, Aggregator)
static <K> CombineFn<K,String>	STRING_CONCAT(String separator, boolean skipNull) Deprecated. Use Aggregators.STRING_CONCAT(String, boolean)
static <K> CombineFn<K,String>	STRING_CONCAT(String separator, boolean skipNull, long maxOutputLength, long maxInputLength) Deprecated. Use Aggregators.STRING_CONCAT(String, boolean, long, long)
static <K> CombineFn<K,BigInteger>	SUM_BIGINTS() Deprecated. Use Aggregators.SUM_BIGINTS()
static <K> CombineFn<K,Double>	SUM_DOUBLES() Deprecated. Use Aggregators.SUM_DOUBLES()
static <K> CombineFn<K,Float>	SUM_FLOATS() Deprecated. Use Aggregators.SUM_FLOATS()
static <K> CombineFn<K,Integer>	SUM_INTS() Deprecated. Use Aggregators.SUM_INTS()
static <K> CombineFn<K,Long>	SUM_LONGS() Deprecated. Use Aggregators.SUM_LONGS()
static <K,A,B,C> CombineFn<K,Tuple3<A,B,C>>	tripAggregator(CombineFn.AggregatorFactory<A> a1, CombineFn.AggregatorFactory<B> a2, CombineFn.AggregatorFactory<C> a3) Deprecated. Use Aggregators.tripAggregator(Aggregator, Aggregator, Aggregator)
static <K> CombineFn<K,TupleN>	tupleAggregator(CombineFn.AggregatorFactory<?>... factories) Deprecated. Use Aggregators.tupleAggregator(Aggregator...)

Methods inherited from class org.apache.crunch.DoFn

cleanup, configure, initialize, process, scaleFactor, setContext

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SUM_LONGS

public static CombineFn.AggregatorFactory<Long> SUM_LONGS

Deprecated. Use Aggregators.SUM_LONGS()

SUM_INTS

public static CombineFn.AggregatorFactory<Integer> SUM_INTS

Deprecated. Use Aggregators.SUM_INTS()

SUM_FLOATS

public static CombineFn.AggregatorFactory<Float> SUM_FLOATS

Deprecated. Use Aggregators.SUM_FLOATS()

SUM_DOUBLES

public static CombineFn.AggregatorFactory<Double> SUM_DOUBLES

Deprecated. Use Aggregators.SUM_DOUBLES()

SUM_BIGINTS

public static CombineFn.AggregatorFactory<BigInteger> SUM_BIGINTS

Deprecated. Use Aggregators.SUM_BIGINTS()

MAX_LONGS

public static CombineFn.AggregatorFactory<Long> MAX_LONGS

Deprecated. Use Aggregators.MAX_LONGS()

MAX_INTS

public static CombineFn.AggregatorFactory<Integer> MAX_INTS

Deprecated. Use Aggregators.MAX_INTS()

MAX_FLOATS

public static CombineFn.AggregatorFactory<Float> MAX_FLOATS

Deprecated. Use Aggregators.MAX_FLOATS()

MAX_DOUBLES

public static CombineFn.AggregatorFactory<Double> MAX_DOUBLES

Deprecated. Use Aggregators.MAX_DOUBLES()

MAX_BIGINTS

public static CombineFn.AggregatorFactory<BigInteger> MAX_BIGINTS

Deprecated. Use Aggregators.MAX_BIGINTS()

MIN_LONGS

public static CombineFn.AggregatorFactory<Long> MIN_LONGS

Deprecated. Use Aggregators.MIN_LONGS()

MIN_INTS

public static CombineFn.AggregatorFactory<Integer> MIN_INTS

Deprecated. Use Aggregators.MIN_INTS()

MIN_FLOATS

public static CombineFn.AggregatorFactory<Float> MIN_FLOATS

Deprecated. Use Aggregators.MIN_FLOATS()

MIN_DOUBLES

public static CombineFn.AggregatorFactory<Double> MIN_DOUBLES

Deprecated. Use Aggregators.MIN_DOUBLES()

MIN_BIGINTS

public static CombineFn.AggregatorFactory<BigInteger> MIN_BIGINTS

Deprecated. Use Aggregators.MIN_BIGINTS()

Constructor Detail

CombineFn

public CombineFn()

Method Detail

aggregator

public static final <K,V> CombineFn<K,V> aggregator(CombineFn.Aggregator<V> aggregator)

Deprecated. Use Aggregators.toCombineFn(Aggregator)

aggregatorFactory

public static final <K,V> CombineFn<K,V> aggregatorFactory(CombineFn.AggregatorFactory<V> aggregator)

Deprecated. Use PGroupedTable.combineValues(Aggregator) which doesn't require a factory.

pairAggregator

public static final <K,V1,V2> CombineFn<K,Pair<V1,V2>> pairAggregator(CombineFn.AggregatorFactory<V1> a1,
                                                                      CombineFn.AggregatorFactory<V2> a2)

Deprecated. Use Aggregators.pairAggregator(Aggregator, Aggregator)

tripAggregator

public static final <K,A,B,C> CombineFn<K,Tuple3<A,B,C>> tripAggregator(CombineFn.AggregatorFactory<A> a1,
                                                                        CombineFn.AggregatorFactory<B> a2,
                                                                        CombineFn.AggregatorFactory<C> a3)

Deprecated. Use Aggregators.tripAggregator(Aggregator, Aggregator, Aggregator)

quadAggregator

public static final <K,A,B,C,D> CombineFn<K,Tuple4<A,B,C,D>> quadAggregator(CombineFn.AggregatorFactory<A> a1,
                                                                            CombineFn.AggregatorFactory<B> a2,
                                                                            CombineFn.AggregatorFactory<C> a3,
                                                                            CombineFn.AggregatorFactory<D> a4)

Deprecated. Use Aggregators.quadAggregator(Aggregator, Aggregator, Aggregator, Aggregator)

tupleAggregator

public static final <K> CombineFn<K,TupleN> tupleAggregator(CombineFn.AggregatorFactory<?>... factories)

Deprecated. Use Aggregators.tupleAggregator(Aggregator...)

SUM_LONGS

public static final <K> CombineFn<K,Long> SUM_LONGS()

Deprecated. Use Aggregators.SUM_LONGS()

SUM_INTS

public static final <K> CombineFn<K,Integer> SUM_INTS()

Deprecated. Use Aggregators.SUM_INTS()

SUM_FLOATS

public static final <K> CombineFn<K,Float> SUM_FLOATS()

Deprecated. Use Aggregators.SUM_FLOATS()

SUM_DOUBLES

public static final <K> CombineFn<K,Double> SUM_DOUBLES()

Deprecated. Use Aggregators.SUM_DOUBLES()

SUM_BIGINTS

public static final <K> CombineFn<K,BigInteger> SUM_BIGINTS()

Deprecated. Use Aggregators.SUM_BIGINTS()

MAX_LONGS

public static final <K> CombineFn<K,Long> MAX_LONGS()

Deprecated. Use Aggregators.MAX_LONGS()

MAX_LONGS

public static final <K> CombineFn<K,Long> MAX_LONGS(int n)

Deprecated. Use Aggregators.MAX_LONGS(int)

MAX_INTS

public static final <K> CombineFn<K,Integer> MAX_INTS()

Deprecated. Use Aggregators.MAX_INTS()

MAX_INTS

public static final <K> CombineFn<K,Integer> MAX_INTS(int n)

Deprecated. Use Aggregators.MAX_INTS(int)

MAX_FLOATS

public static final <K> CombineFn<K,Float> MAX_FLOATS()

Deprecated. Use Aggregators.MAX_FLOATS()

MAX_FLOATS

public static final <K> CombineFn<K,Float> MAX_FLOATS(int n)

Deprecated. Use Aggregators.MAX_FLOATS(int)

MAX_DOUBLES

public static final <K> CombineFn<K,Double> MAX_DOUBLES()

Deprecated. Use Aggregators.MAX_DOUBLES()

MAX_DOUBLES

public static final <K> CombineFn<K,Double> MAX_DOUBLES(int n)

Deprecated. Use Aggregators.MAX_DOUBLES(int)

MAX_BIGINTS

public static final <K> CombineFn<K,BigInteger> MAX_BIGINTS()

Deprecated. Use Aggregators.MAX_BIGINTS()

MAX_BIGINTS

public static final <K> CombineFn<K,BigInteger> MAX_BIGINTS(int n)

Deprecated. Use Aggregators.MAX_BIGINTS(int)

MIN_LONGS

public static final <K> CombineFn<K,Long> MIN_LONGS()

Deprecated. Use Aggregators.MIN_LONGS()

MIN_LONGS

public static final <K> CombineFn<K,Long> MIN_LONGS(int n)

Deprecated. Use Aggregators.MIN_LONGS(int)

MIN_INTS

public static final <K> CombineFn<K,Integer> MIN_INTS()

Deprecated. Use Aggregators.MIN_INTS()

MIN_INTS

public static final <K> CombineFn<K,Integer> MIN_INTS(int n)

Deprecated. Use Aggregators.MIN_INTS(int)

MIN_FLOATS

public static final <K> CombineFn<K,Float> MIN_FLOATS()

Deprecated. Use Aggregators.MIN_FLOATS()

MIN_FLOATS

public static final <K> CombineFn<K,Float> MIN_FLOATS(int n)

Deprecated. Use Aggregators.MIN_FLOATS(int)

MIN_DOUBLES

public static final <K> CombineFn<K,Double> MIN_DOUBLES()

Deprecated. Use Aggregators.MIN_DOUBLES()

MIN_DOUBLES

public static final <K> CombineFn<K,Double> MIN_DOUBLES(int n)

Deprecated. Use Aggregators.MIN_DOUBLES(int)

MIN_BIGINTS

public static final <K> CombineFn<K,BigInteger> MIN_BIGINTS()

Deprecated. Use Aggregators.MIN_BIGINTS()

MIN_BIGINTS

public static final <K> CombineFn<K,BigInteger> MIN_BIGINTS(int n)

Deprecated. Use Aggregators.MIN_BIGINTS(int)

FIRST_N

public static final <K,V> CombineFn<K,V> FIRST_N(int n)

Deprecated. Use Aggregators.FIRST_N(int)

LAST_N

public static final <K,V> CombineFn<K,V> LAST_N(int n)

Deprecated. Use Aggregators.LAST_N(int)

STRING_CONCAT

public static final <K> CombineFn<K,String> STRING_CONCAT(String separator,
                                                          boolean skipNull)

Deprecated. Use Aggregators.STRING_CONCAT(String, boolean)

Used to concatenate strings, with a separator between each strings. There is no limits of length for the concatenated string.

Parameters:

separator - the separator which will be appended between each string

skipNull - define if we should skip null values. Throw NullPointerException if set to false and there is a null value.

Returns:

STRING_CONCAT

public static final <K> CombineFn<K,String> STRING_CONCAT(String separator,
                                                          boolean skipNull,
                                                          long maxOutputLength,
                                                          long maxInputLength)

Deprecated. Use Aggregators.STRING_CONCAT(String, boolean, long, long)

Used to concatenate strings, with a separator between each strings. You can specify the maximum length of the output string and of the input strings, if they are > 0. If a value is <= 0, there is no limits. Any too large string (or any string which would made the output too large) will be silently discarded.

Parameters:

separator - the separator which will be appended between each string

skipNull - define if we should skip null values. Throw NullPointerException if set to false and there is a null value.

maxOutputLength - the maximum length of the output string. If it's set <= 0, there is no limits. The number of characters of the output string will be < maxOutputLength.

maxInputLength - the maximum length of the input strings. If it's set <= 0, there is no limits. The number of characters of the int string will be < maxInputLength to be concatenated.

Returns: