This project has retired. For details please refer to its Attic page.

org.apache.crunch.lib
Class Shard

java.lang.Object
  org.apache.crunch.lib.Shard

public class Shard
extends Object

Utilities for controlling how the data in a PCollection is balanced across reducers and output files.

Constructor Summary

Shard()

Method Summary

static <T> PCollection<T>

shard(PCollection<T> pc, int numPartitions) Creates a PCollection<T> that has the same contents as its input argument but will be written to a fixed number of output files.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Shard

public Shard()

Method Detail

shard

public static <T> PCollection<T> shard(PCollection<T> pc,
                                       int numPartitions)

Creates a PCollection<T> that has the same contents as its input argument but will be written to a fixed number of output files. This is useful for map-only jobs that process lots of input files but only write out a small amount of input per task.

Parameters:

pc - The PCollection<T> to rebalance

numPartitions - The number of output partitions to create

Returns:

A rebalanced PCollection<T> with the same contents as the input