org.apache.crunch.util

Class PartitionUtils

java.lang.Object

org.apache.crunch.util.PartitionUtils

public class PartitionUtils
extends Object

Helper functions and settings for determining the number of reducers to use in a pipeline job created by the Crunch planner.

Field Summary

Fields

Modifier and Type	Field and Description
static String	BYTES_PER_REDUCE_TASK
static long	DEFAULT_BYTES_PER_REDUCE_TASK
static int	DEFAULT_MAX_REDUCERS
static String	MAX_REDUCERS Set an upper limit on the number of reducers the Crunch planner will set for an MR job when it tries to determine how many reducers to use based on the input size.

Constructor Summary

Constructors

Constructor and Description
PartitionUtils()

Method Summary

Modifier and Type	Method and Description
static <T> int	getRecommendedPartitions(PCollection<T> pcollection)
static <T> int	getRecommendedPartitions(PCollection<T> pcollection, org.apache.hadoop.conf.Configuration conf)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BYTES_PER_REDUCE_TASK

public static final String BYTES_PER_REDUCE_TASK

See Also:

Constant Field Values

DEFAULT_BYTES_PER_REDUCE_TASK

public static final long DEFAULT_BYTES_PER_REDUCE_TASK

See Also:

Constant Field Values

MAX_REDUCERS

public static final String MAX_REDUCERS

Set an upper limit on the number of reducers the Crunch planner will set for an MR job when it tries to determine how many reducers to use based on the input size.

See Also:

Constant Field Values

DEFAULT_MAX_REDUCERS

public static final int DEFAULT_MAX_REDUCERS

See Also:

Constant Field Values

Constructor Detail

PartitionUtils

public PartitionUtils()

Method Detail

getRecommendedPartitions

public static <T> int getRecommendedPartitions(PCollection<T> pcollection)

getRecommendedPartitions

public static <T> int getRecommendedPartitions(PCollection<T> pcollection,
                                               org.apache.hadoop.conf.Configuration conf)