Project Crunch has retired. For details please refer to its Attic page.
To (Apache Crunch 0.10.0 API)

org.apache.crunch.io
Class To

java.lang.Object
  extended by org.apache.crunch.io.To

public class To
extends Object

Static factory methods for creating common Target types.

The To class is intended to be used as part of a literate API for writing the output of Crunch pipelines to common file types. We can use the Target objects created by the factory methods in the To class with either the write method on the Pipeline class or the convenience write method on PCollection and PTable instances. Pipeline pipeline = new MRPipeline(this.getClass()); ... // Write a PCollection to a text file: PCollection words = ...; pipeline.write(words, To.textFile("/put/my/words/here")); // Write a PTable to a sequence file: PTable textToText = ...; textToText.write(To.sequenceFile("/words/to/words")); // Write a PCollection to an Avro data file: PCollection objects = ...; objects.write(To.avroFile("/my/avro/files")); // Write a PTable to a custom FileOutputFormat: PTable custom = ...; pipeline.write(custom, To.formattedFile("/custom", MyFileFormat.class));


Constructor Summary
To()
           
 
Method Summary
static Target avroFile(org.apache.hadoop.fs.Path path)
          Creates a Target at the given Path that writes data to Avro files.
static Target avroFile(String pathName)
          Creates a Target at the given path name that writes data to Avro files.
static
<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>
Target
formattedFile(org.apache.hadoop.fs.Path path, Class<? extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<K,V>> formatClass)
          Creates a Target at the given Path that writes data to a custom FileOutputFormat.
static
<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>
Target
formattedFile(String pathName, Class<? extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<K,V>> formatClass)
          Creates a Target at the given path name that writes data to a custom FileOutputFormat.
static Target sequenceFile(org.apache.hadoop.fs.Path path)
          Creates a Target at the given Path that writes data to SequenceFiles.
static Target sequenceFile(String pathName)
          Creates a Target at the given path name that writes data to SequenceFiles.
static Target textFile(org.apache.hadoop.fs.Path path)
          Creates a Target at the given Path that writes data to text files.
static Target textFile(String pathName)
          Creates a Target at the given path name that writes data to text files.
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

To

public To()
Method Detail

formattedFile

public static <K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable> Target formattedFile(String pathName,
                                                                                                                     Class<? extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<K,V>> formatClass)
Creates a Target at the given path name that writes data to a custom FileOutputFormat.

Parameters:
pathName - The name of the path to write the data to on the filesystem
formatClass - The FileOutputFormat<K, V> to write the data to
Returns:
A new Target instance

formattedFile

public static <K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable> Target formattedFile(org.apache.hadoop.fs.Path path,
                                                                                                                     Class<? extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<K,V>> formatClass)
Creates a Target at the given Path that writes data to a custom FileOutputFormat.

Parameters:
path - The Path to write the data to
formatClass - The FileOutputFormat to write the data to
Returns:
A new Target instance

avroFile

public static Target avroFile(String pathName)
Creates a Target at the given path name that writes data to Avro files. The PType for the written data must be for Avro records.

Parameters:
pathName - The name of the path to write the data to on the filesystem
Returns:
A new Target instance

avroFile

public static Target avroFile(org.apache.hadoop.fs.Path path)
Creates a Target at the given Path that writes data to Avro files. The PType for the written data must be for Avro records.

Parameters:
path - The Path to write the data to
Returns:
A new Target instance

sequenceFile

public static Target sequenceFile(String pathName)
Creates a Target at the given path name that writes data to SequenceFiles.

Parameters:
pathName - The name of the path to write the data to on the filesystem
Returns:
A new Target instance

sequenceFile

public static Target sequenceFile(org.apache.hadoop.fs.Path path)
Creates a Target at the given Path that writes data to SequenceFiles.

Parameters:
path - The Path to write the data to
Returns:
A new Target instance

textFile

public static Target textFile(String pathName)
Creates a Target at the given path name that writes data to text files.

Parameters:
pathName - The name of the path to write the data to on the filesystem
Returns:
A new Target instance

textFile

public static Target textFile(org.apache.hadoop.fs.Path path)
Creates a Target at the given Path that writes data to text files.

Parameters:
path - The Path to write the data to
Returns:
A new Target instance


Copyright © 2014 The Apache Software Foundation. All Rights Reserved.