MemPipeline (Apache Crunch 0.3.0-incubating API)

This project has retired. For details please refer to its Attic page.

Overview

Package

Class

Use

Tree

Deprecated

Index

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

org.apache.crunch.impl.mem
Class MemPipeline

java.lang.Object
  org.apache.crunch.impl.mem.MemPipeline

All Implemented Interfaces:: Pipeline

public class MemPipeline
extends Object
implements Pipeline
extends Object
implements Pipeline

Method Summary

static <T> PCollection<T> collectionOf(Iterable<T> collect)


static <T> PCollection<T> collectionOf(T... ts)


PipelineResult done()
          Run any remaining jobs required to generate outputs and then clean up any intermediate data files that were created in this run or previous calls to run.

void enableDebug()
          Turn on debug logging for jobs that are run from this pipeline.

org.apache.hadoop.conf.Configuration getConfiguration()
          Returns the Configuration instance associated with this pipeline.

static Pipeline getInstance()


String getName()
          Returns the name of this pipeline.

<T> Iterable<T> materialize(PCollection<T> pcollection)
          Create the given PCollection and read the data it contains into the returned Collection instance for client use.

<T> PCollection<T> read(Source<T> source)
          Converts the given Source into a PCollection that is available to jobs run using this Pipeline instance.

<K,V> PTable<K,V> read(TableSource<K,V> source)
          A version of the read method for TableSource instances that map to PTables.

PCollection<String> readTextFile(String pathName)
          A convenience method for reading a text file.

PipelineResult run()
          Constructs and executes a series of MapReduce jobs in order to write data to the output targets.

void setConfiguration(org.apache.hadoop.conf.Configuration conf)
          Set the Configuration to use with this pipeline.

static <S,T> PTable<S,T> tableOf(Iterable<Pair<S,T>> pairs)


static <S,T> PTable<S,T> tableOf(S s, T t, Object... more)


static <T> PCollection<T> typedCollectionOf(PType<T> ptype, Iterable<T> collect)


static <T> PCollection<T> typedCollectionOf(PType<T> ptype, T... ts)


static <S,T> PTable<S,T> typedTableOf(PTableType<S,T> ptype, Iterable<Pair<S,T>> pairs)


static <S,T> PTable<S,T> typedTableOf(PTableType<S,T> ptype, S s, T t, Object... more)


void write(PCollection<?> collection, Target target)
          Write the given collection to the given target on the next pipeline run.

<T> void writeTextFile(PCollection<T> collection, String pathName)
          A convenience method for writing a text file.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from class java.lang.Object
`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Method Detail

getInstance

public static Pipeline getInstance()

collectionOf

public static <T> PCollection<T> collectionOf(T... ts)

collectionOf

public static <T> PCollection<T> collectionOf(Iterable<T> collect)

typedCollectionOf

public static <T> PCollection<T> typedCollectionOf(PType<T> ptype,
                                                   T... ts)

typedCollectionOf

public static <T> PCollection<T> typedCollectionOf(PType<T> ptype,
                                                   Iterable<T> collect)

tableOf

public static <S,T> PTable<S,T> tableOf(S s,
                                        T t,
                                        Object... more)

typedTableOf

public static <S,T> PTable<S,T> typedTableOf(PTableType<S,T> ptype,
                                             S s,
                                             T t,
                                             Object... more)

tableOf

public static <S,T> PTable<S,T> tableOf(Iterable<Pair<S,T>> pairs)