org.apache.crunch.io

Class CrunchOutputs<K,V>

java.lang.Object

org.apache.crunch.io.CrunchOutputs<K,V>

public class CrunchOutputs<K,V>
extends Object

An analogue of CrunchInputs for handling multiple OutputFormat instances writing to multiple files within a single MapReduce job.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CrunchOutputs.OutputConfig<K,V>

Field Summary

Fields

Modifier and Type	Field and Description
static String	CRUNCH_DISABLE_OUTPUT_COUNTERS
static String	CRUNCH_OUTPUTS

Constructor Summary

Constructors

Constructor and Description
CrunchOutputs(org.apache.hadoop.conf.Configuration conf)
CrunchOutputs(org.apache.hadoop.mapreduce.TaskInputOutputContext<?,?,K,V> context) Creates and initializes multiple outputs support, it should be instantiated in the Mapper/Reducer setup method.

Method Summary

Methods

Modifier and Type	Method and Description
static void	addNamedOutput(org.apache.hadoop.mapreduce.Job job, String name, Class<? extends org.apache.hadoop.mapreduce.OutputFormat> outputFormatClass, Class keyClass, Class valueClass)
static void	addNamedOutput(org.apache.hadoop.mapreduce.Job job, String name, FormatBundle<? extends org.apache.hadoop.mapreduce.OutputFormat> outputBundle, Class keyClass, Class valueClass)
static void	checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext jc)
void	close()
static Map<String,CrunchOutputs.OutputConfig>	getNamedOutputs(org.apache.hadoop.conf.Configuration conf)
static org.apache.hadoop.mapreduce.OutputCommitter	getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext tac)
void	write(String namedOutput, K key, V value)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CRUNCH_OUTPUTS

public static final String CRUNCH_OUTPUTS

See Also:

Constant Field Values

CRUNCH_DISABLE_OUTPUT_COUNTERS

public static final String CRUNCH_DISABLE_OUTPUT_COUNTERS

See Also:

Constant Field Values

Constructor Detail

CrunchOutputs

public CrunchOutputs(org.apache.hadoop.mapreduce.TaskInputOutputContext<?,?,K,V> context)

Creates and initializes multiple outputs support, it should be instantiated in the Mapper/Reducer setup method.

Parameters:

context - the TaskInputOutputContext object

CrunchOutputs

public CrunchOutputs(org.apache.hadoop.conf.Configuration conf)

Method Detail

addNamedOutput

public static void addNamedOutput(org.apache.hadoop.mapreduce.Job job,
                  String name,
                  Class<? extends org.apache.hadoop.mapreduce.OutputFormat> outputFormatClass,
                  Class keyClass,
                  Class valueClass)

addNamedOutput

public static void addNamedOutput(org.apache.hadoop.mapreduce.Job job,
                  String name,
                  FormatBundle<? extends org.apache.hadoop.mapreduce.OutputFormat> outputBundle,
                  Class keyClass,
                  Class valueClass)

checkOutputSpecs

public static void checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext jc)
                             throws IOException,
                                    InterruptedException

Throws:

IOException

InterruptedException

getOutputCommitter

public static org.apache.hadoop.mapreduce.OutputCommitter getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext tac)
                                                                      throws IOException,
                                                                             InterruptedException

Throws:

IOException

InterruptedException

getNamedOutputs

public static Map<String,CrunchOutputs.OutputConfig> getNamedOutputs(org.apache.hadoop.conf.Configuration conf)

write

public void write(String namedOutput,
         K key,
         V value)
           throws IOException,
                  InterruptedException

Throws:

IOException

InterruptedException

close

public void close()
           throws IOException,
                  InterruptedException

Throws:

IOException

InterruptedException