This project has retired. For details please refer to its Attic page.
PartitionUtils (Apache Crunch 0.10.0 API)

org.apache.crunch.util
Class PartitionUtils

java.lang.Object
  extended by org.apache.crunch.util.PartitionUtils

public class PartitionUtils
extends Object

Helper functions and settings for determining the number of reducers to use in a pipeline job created by the Crunch planner.


Field Summary
static String BYTES_PER_REDUCE_TASK
           
static long DEFAULT_BYTES_PER_REDUCE_TASK
           
static int DEFAULT_MAX_REDUCERS
           
static String MAX_REDUCERS
          Set an upper limit on the number of reducers the Crunch planner will set for an MR job when it tries to determine how many reducers to use based on the input size.
 
Constructor Summary
PartitionUtils()
           
 
Method Summary
static
<T> int
getRecommendedPartitions(PCollection<T> pcollection)
           
static
<T> int
getRecommendedPartitions(PCollection<T> pcollection, org.apache.hadoop.conf.Configuration conf)
           
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

BYTES_PER_REDUCE_TASK

public static final String BYTES_PER_REDUCE_TASK
See Also:
Constant Field Values

DEFAULT_BYTES_PER_REDUCE_TASK

public static final long DEFAULT_BYTES_PER_REDUCE_TASK
See Also:
Constant Field Values

MAX_REDUCERS

public static final String MAX_REDUCERS
Set an upper limit on the number of reducers the Crunch planner will set for an MR job when it tries to determine how many reducers to use based on the input size.

See Also:
Constant Field Values

DEFAULT_MAX_REDUCERS

public static final int DEFAULT_MAX_REDUCERS
See Also:
Constant Field Values
Constructor Detail

PartitionUtils

public PartitionUtils()
Method Detail

getRecommendedPartitions

public static <T> int getRecommendedPartitions(PCollection<T> pcollection)

getRecommendedPartitions

public static <T> int getRecommendedPartitions(PCollection<T> pcollection,
                                               org.apache.hadoop.conf.Configuration conf)


Copyright © 2014 The Apache Software Foundation. All Rights Reserved.