|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.apache.crunch.impl.mem.MemPipeline
public class MemPipeline
Method Summary | ||
---|---|---|
static void |
clearCounters()
|
|
static
|
collectionOf(Iterable<T> collect)
|
|
static
|
collectionOf(T... ts)
|
|
PipelineResult |
done()
Run any remaining jobs required to generate outputs and then clean up any intermediate data files that were created in this run or previous calls to run . |
|
void |
enableDebug()
Turn on debug logging for jobs that are run from this pipeline. |
|
org.apache.hadoop.conf.Configuration |
getConfiguration()
Returns the Configuration instance associated with this pipeline. |
|
static org.apache.hadoop.mapreduce.Counters |
getCounters()
|
|
static Pipeline |
getInstance()
|
|
String |
getName()
Returns the name of this pipeline. |
|
|
materialize(PCollection<T> pcollection)
Create the given PCollection and read the data it contains into the returned Collection instance for client use. |
|
|
read(Source<T> source)
Converts the given Source into a PCollection that is
available to jobs run using this Pipeline instance. |
|
|
read(TableSource<K,V> source)
A version of the read method for TableSource instances that map to
PTable s. |
|
PCollection<String> |
readTextFile(String pathName)
A convenience method for reading a text file. |
|
PipelineResult |
run()
Constructs and executes a series of MapReduce jobs in order to write data to the output targets. |
|
PipelineExecution |
runAsync()
Constructs and starts a series of MapReduce jobs in order ot write data to the output targets, but returns a ListenableFuture to allow clients to control
job execution. |
|
void |
setConfiguration(org.apache.hadoop.conf.Configuration conf)
Set the Configuration to use with this pipeline. |
|
static
|
tableOf(Iterable<Pair<S,T>> pairs)
|
|
static
|
tableOf(S s,
T t,
Object... more)
|
|
static
|
typedCollectionOf(PType<T> ptype,
Iterable<T> collect)
|
|
static
|
typedCollectionOf(PType<T> ptype,
T... ts)
|
|
static
|
typedTableOf(PTableType<S,T> ptype,
Iterable<Pair<S,T>> pairs)
|
|
static
|
typedTableOf(PTableType<S,T> ptype,
S s,
T t,
Object... more)
|
|
void |
write(PCollection<?> collection,
Target target)
Write the given collection to the given target on the next pipeline run. |
|
void |
write(PCollection<?> collection,
Target target,
Target.WriteMode writeMode)
Write the contents of the PCollection to the given Target ,
using the storage format specified by the target and the given
WriteMode for cases where the referenced Target
already exists. |
|
|
writeTextFile(PCollection<T> collection,
String pathName)
A convenience method for writing a text file. |
Methods inherited from class java.lang.Object |
---|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Method Detail |
---|
public static org.apache.hadoop.mapreduce.Counters getCounters()
public static void clearCounters()
public static Pipeline getInstance()
public static <T> PCollection<T> collectionOf(T... ts)
public static <T> PCollection<T> collectionOf(Iterable<T> collect)
public static <T> PCollection<T> typedCollectionOf(PType<T> ptype, T... ts)
public static <T> PCollection<T> typedCollectionOf(PType<T> ptype, Iterable<T> collect)
public static <S,T> PTable<S,T> tableOf(S s, T t, Object... more)
public static <S,T> PTable<S,T> typedTableOf(PTableType<S,T> ptype, S s, T t, Object... more)
public static <S,T> PTable<S,T> tableOf(Iterable<Pair<S,T>> pairs)
public static <S,T> PTable<S,T> typedTableOf(PTableType<S,T> ptype, Iterable<Pair<S,T>> pairs)
public void setConfiguration(org.apache.hadoop.conf.Configuration conf)
Pipeline
Configuration
to use with this pipeline.
setConfiguration
in interface Pipeline
public org.apache.hadoop.conf.Configuration getConfiguration()
Pipeline
Configuration
instance associated with this pipeline.
getConfiguration
in interface Pipeline
public <T> PCollection<T> read(Source<T> source)
Pipeline
Source
into a PCollection
that is
available to jobs run using this Pipeline
instance.
read
in interface Pipeline
source
- The source of data
public <K,V> PTable<K,V> read(TableSource<K,V> source)
Pipeline
TableSource
instances that map to
PTable
s.
read
in interface Pipeline
source
- The source of the data
public void write(PCollection<?> collection, Target target)
Pipeline
WriteMode.DEFAULT
rule for the given Target
.
write
in interface Pipeline
collection
- The collectiontarget
- The output targetpublic void write(PCollection<?> collection, Target target, Target.WriteMode writeMode)
Pipeline
PCollection
to the given Target
,
using the storage format specified by the target and the given
WriteMode
for cases where the referenced Target
already exists.
write
in interface Pipeline
collection
- The collectiontarget
- The target to write towriteMode
- The strategy to use for handling existing outputspublic PCollection<String> readTextFile(String pathName)
Pipeline
readTextFile
in interface Pipeline
public <T> void writeTextFile(PCollection<T> collection, String pathName)
Pipeline
writeTextFile
in interface Pipeline
public <T> Iterable<T> materialize(PCollection<T> pcollection)
Pipeline
materialize
in interface Pipeline
pcollection
- The PCollection to materialize
public PipelineExecution runAsync()
Pipeline
ListenableFuture
to allow clients to control
job execution.
runAsync
in interface Pipeline
public PipelineResult run()
Pipeline
run
in interface Pipeline
public PipelineResult done()
Pipeline
run
.
done
in interface Pipeline
public void enableDebug()
Pipeline
enableDebug
in interface Pipeline
public String getName()
Pipeline
getName
in interface Pipeline
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |