org.apache.crunch.lib

Class Channels

java.lang.Object

org.apache.crunch.lib.Channels

public class Channels
extends Object

Utilities for splitting Pair instances emitted by DoFn into separate PCollection instances. A typical motivation for this might be to separate standard output from error output of a DoFn.

Author:

Brandon Inman

Constructor Summary

Constructors

Constructor and Description
Channels()

Method Summary

Methods

Modifier and Type	Method and Description
static <T,U> Pair<PCollection<T>,PCollection<U>>	split(PCollection<Pair<T,U>> pCollection) Splits a PCollection of any Pair of objects into a Pair of PCollection}, to allow for the output of a DoFn to be handled using separate channels.
static <T,U> Pair<PCollection<T>,PCollection<U>>	split(PCollection<Pair<T,U>> pCollection, PType<T> firstPType, PType<U> secondPType) Splits a PCollection of any Pair of objects into a Pair of PCollection}, to allow for the output of a DoFn to be handled using separate channels.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Channels

public Channels()

Method Detail

split

public static <T,U> Pair<PCollection<T>,PCollection<U>> split(PCollection<Pair<T,U>> pCollection)

Splits a PCollection of any Pair of objects into a Pair of PCollection}, to allow for the output of a DoFn to be handled using separate channels.

Parameters:

pCollection - The PCollection to split

split

public static <T,U> Pair<PCollection<T>,PCollection<U>> split(PCollection<Pair<T,U>> pCollection,
                                              PType<T> firstPType,
                                              PType<U> secondPType)

Splits a PCollection of any Pair of objects into a Pair of PCollection}, to allow for the output of a DoFn to be handled using separate channels.

Parameters:

pCollection - The PCollection to split

firstPType - The PType for the first collection

secondPType - The PType for the second collection

Returns:

Pair of PCollection