|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.apache.crunch.impl.dist.collect.PCollectionImpl<S>
public abstract class PCollectionImpl<S>
Nested Class Summary | |
---|---|
static interface |
PCollectionImpl.Visitor
|
Field Summary | |
---|---|
protected ParallelDoOptions |
doOptions
|
protected SourceTarget<S> |
materializedAt
|
protected DistributedPipeline |
pipeline
|
Constructor Summary | |
---|---|
PCollectionImpl(String name,
DistributedPipeline pipeline)
|
|
PCollectionImpl(String name,
DistributedPipeline pipeline,
ParallelDoOptions doOptions)
|
Method Summary | ||
---|---|---|
void |
accept(PCollectionImpl.Visitor visitor)
|
|
protected abstract void |
acceptInternal(PCollectionImpl.Visitor visitor)
|
|
PObject<Collection<S>> |
asCollection()
|
|
ReadableData<S> |
asReadable(boolean materialize)
|
|
|
by(MapFn<S,K> mapFn,
PType<K> keyType)
Apply the given map function to each element of this instance in order to create a PTable . |
|
|
by(String name,
MapFn<S,K> mapFn,
PType<K> keyType)
Apply the given map function to each element of this instance in order to create a PTable . |
|
PCollection<S> |
cache()
Marks this data as cached using the default CachingOptions . |
|
PCollection<S> |
cache(CachingOptions options)
Marks this data as cached using the given CachingOptions . |
|
PTable<S,Long> |
count()
Returns a PTable instance that contains the counts of each unique
element of this PCollection. |
|
PCollection<S> |
filter(FilterFn<S> filterFn)
Apply the given filter function to this instance and return the resulting PCollection . |
|
PCollection<S> |
filter(String name,
FilterFn<S> filterFn)
Apply the given filter function to this instance and return the resulting PCollection . |
|
protected PCollectionImpl<S> |
getChainingCollection()
Retrieve the PCollectionImpl to be used for chaining within PCollectionImpls further down the pipeline. |
|
int |
getDepth()
|
|
abstract long |
getLastModifiedAt()
|
|
SourceTarget<S> |
getMaterializedAt()
|
|
String |
getName()
Returns a shorthand name for this PCollection. |
|
PCollectionImpl<?> |
getOnlyParent()
|
|
ParallelDoOptions |
getParallelDoOptions()
|
|
abstract List<PCollectionImpl<?>> |
getParents()
|
|
DistributedPipeline |
getPipeline()
Returns the Pipeline associated with this PCollection. |
|
protected abstract ReadableData<S> |
getReadableDataInternal()
|
|
long |
getSize()
Returns the size of the data represented by this PCollection in
bytes. |
|
protected abstract long |
getSizeInternal()
|
|
Set<SourceTarget<?>> |
getTargetDependencies()
|
|
PTypeFamily |
getTypeFamily()
Returns the PTypeFamily of this PCollection . |
|
boolean |
isBreakpoint()
|
|
PObject<Long> |
length()
Returns the number of elements represented by this PCollection . |
|
Iterable<S> |
materialize()
Returns a reference to the data set represented by this PCollection that may be used by the client to read the data locally. |
|
void |
materializeAt(SourceTarget<S> sourceTarget)
|
|
protected ReadableData<S> |
materializedData()
|
|
PObject<S> |
max()
Returns a PObject of the maximum element of this instance. |
|
PObject<S> |
min()
Returns a PObject of the minimum element of this instance. |
|
|
parallelDo(DoFn<S,Pair<K,V>> fn,
PTableType<K,V> type)
Similar to the other parallelDo instance, but returns a
PTable instance instead of a PCollection . |
|
|
parallelDo(DoFn<S,T> fn,
PType<T> type)
Applies the given doFn to the elements of this PCollection and
returns a new PCollection that is the output of this processing. |
|
|
parallelDo(String name,
DoFn<S,Pair<K,V>> fn,
PTableType<K,V> type)
Similar to the other parallelDo instance, but returns a
PTable instance instead of a PCollection . |
|
|
parallelDo(String name,
DoFn<S,Pair<K,V>> fn,
PTableType<K,V> type,
ParallelDoOptions options)
Similar to the other parallelDo instance, but returns a
PTable instance instead of a PCollection . |
|
|
parallelDo(String name,
DoFn<S,T> fn,
PType<T> type)
Applies the given doFn to the elements of this PCollection and
returns a new PCollection that is the output of this processing. |
|
|
parallelDo(String name,
DoFn<S,T> fn,
PType<T> type,
ParallelDoOptions options)
Applies the given doFn to the elements of this PCollection and
returns a new PCollection that is the output of this processing. |
|
void |
setBreakpoint()
|
|
String |
toString()
|
|
PCollection<S> |
union(PCollection<S>... collections)
Returns a PCollection instance that acts as the union of this
PCollection and the input PCollection s. |
|
PCollection<S> |
union(PCollection<S> other)
Returns a PCollection instance that acts as the union of this
PCollection and the given PCollection . |
|
PCollection<S> |
write(Target target)
Write the contents of this PCollection to the given Target ,
using the storage format specified by the target. |
|
PCollection<S> |
write(Target target,
Target.WriteMode writeMode)
Write the contents of this PCollection to the given Target ,
using the given Target.WriteMode to handle existing
targets. |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait |
Methods inherited from interface org.apache.crunch.PCollection |
---|
getPType |
Field Detail |
---|
protected DistributedPipeline pipeline
protected SourceTarget<S> materializedAt
protected final ParallelDoOptions doOptions
Constructor Detail |
---|
public PCollectionImpl(String name, DistributedPipeline pipeline)
public PCollectionImpl(String name, DistributedPipeline pipeline, ParallelDoOptions doOptions)
Method Detail |
---|
public String getName()
PCollection
getName
in interface PCollection<S>
public DistributedPipeline getPipeline()
PCollection
Pipeline
associated with this PCollection.
getPipeline
in interface PCollection<S>
public ParallelDoOptions getParallelDoOptions()
public String toString()
toString
in class Object
public Iterable<S> materialize()
PCollection
materialize
in interface PCollection<S>
public PCollection<S> cache()
PCollection
CachingOptions
. Cached PCollection
s will only
be processed once, and then their contents will be saved so that downstream code can process them many times.
cache
in interface PCollection<S>
PCollection
instancepublic PCollection<S> cache(CachingOptions options)
PCollection
CachingOptions
. Cached PCollection
s will only
be processed once and then their contents will be saved so that downstream code can process them many times.
cache
in interface PCollection<S>
options
- the options that control the cache settings for the data
PCollection
instancepublic PCollection<S> union(PCollection<S> other)
PCollection
PCollection
instance that acts as the union of this
PCollection
and the given PCollection
.
union
in interface PCollection<S>
public PCollection<S> union(PCollection<S>... collections)
PCollection
PCollection
instance that acts as the union of this
PCollection
and the input PCollection
s.
union
in interface PCollection<S>
public <T> PCollection<T> parallelDo(DoFn<S,T> fn, PType<T> type)
PCollection
PCollection
and
returns a new PCollection
that is the output of this processing.
parallelDo
in interface PCollection<S>
fn
- The DoFn
to applytype
- The PType
of the resulting PCollection
PCollection
public <T> PCollection<T> parallelDo(String name, DoFn<S,T> fn, PType<T> type)
PCollection
PCollection
and
returns a new PCollection
that is the output of this processing.
parallelDo
in interface PCollection<S>
name
- An identifier for this processing step, useful for debuggingfn
- The DoFn
to applytype
- The PType
of the resulting PCollection
PCollection
public <T> PCollection<T> parallelDo(String name, DoFn<S,T> fn, PType<T> type, ParallelDoOptions options)
PCollection
PCollection
and
returns a new PCollection
that is the output of this processing.
parallelDo
in interface PCollection<S>
name
- An identifier for this processing step, useful for debuggingfn
- The DoFn
to applytype
- The PType
of the resulting PCollection
options
- Optional information that is needed for certain pipeline operations
PCollection
public <K,V> PTable<K,V> parallelDo(DoFn<S,Pair<K,V>> fn, PTableType<K,V> type)
PCollection
parallelDo
instance, but returns a
PTable
instance instead of a PCollection
.
parallelDo
in interface PCollection<S>
fn
- The DoFn
to applytype
- The PTableType
of the resulting PTable
PTable
public <K,V> PTable<K,V> parallelDo(String name, DoFn<S,Pair<K,V>> fn, PTableType<K,V> type)
PCollection
parallelDo
instance, but returns a
PTable
instance instead of a PCollection
.
parallelDo
in interface PCollection<S>
name
- An identifier for this processing stepfn
- The DoFn
to applytype
- The PTableType
of the resulting PTable
PTable
public <K,V> PTable<K,V> parallelDo(String name, DoFn<S,Pair<K,V>> fn, PTableType<K,V> type, ParallelDoOptions options)
PCollection
parallelDo
instance, but returns a
PTable
instance instead of a PCollection
.
parallelDo
in interface PCollection<S>
name
- An identifier for this processing stepfn
- The DoFn
to applytype
- The PTableType
of the resulting PTable
options
- Optional information that is needed for certain pipeline operations
PTable
public PCollection<S> write(Target target)
PCollection
PCollection
to the given Target
,
using the storage format specified by the target.
write
in interface PCollection<S>
target
- The target to write topublic PCollection<S> write(Target target, Target.WriteMode writeMode)
PCollection
PCollection
to the given Target
,
using the given Target.WriteMode
to handle existing
targets.
write
in interface PCollection<S>
target
- The targetwriteMode
- The rule for handling existing outputs at the target locationpublic void accept(PCollectionImpl.Visitor visitor)
protected abstract void acceptInternal(PCollectionImpl.Visitor visitor)
public void setBreakpoint()
public boolean isBreakpoint()
public PObject<Collection<S>> asCollection()
asCollection
in interface PCollection<S>
PObject
encapsulating an in-memory Collection
containing the values
of this PCollection
.public SourceTarget<S> getMaterializedAt()
public void materializeAt(SourceTarget<S> sourceTarget)
public PCollection<S> filter(FilterFn<S> filterFn)
PCollection
PCollection
.
filter
in interface PCollection<S>
public PCollection<S> filter(String name, FilterFn<S> filterFn)
PCollection
PCollection
.
filter
in interface PCollection<S>
name
- An identifier for this processing stepfilterFn
- The FilterFn
to applypublic <K> PTable<K,S> by(MapFn<S,K> mapFn, PType<K> keyType)
PCollection
PTable
.
by
in interface PCollection<S>
public <K> PTable<K,S> by(String name, MapFn<S,K> mapFn, PType<K> keyType)
PCollection
PTable
.
by
in interface PCollection<S>
name
- An identifier for this processing stepmapFn
- The MapFn
to applypublic PTable<S,Long> count()
PCollection
PTable
instance that contains the counts of each unique
element of this PCollection.
count
in interface PCollection<S>
public PObject<Long> length()
PCollection
PCollection
.
length
in interface PCollection<S>
PObject
containing the number of elements in this PCollection
.public PObject<S> max()
PCollection
PObject
of the maximum element of this instance.
max
in interface PCollection<S>
public PObject<S> min()
PCollection
PObject
of the minimum element of this instance.
min
in interface PCollection<S>
public PTypeFamily getTypeFamily()
PCollection
PTypeFamily
of this PCollection
.
getTypeFamily
in interface PCollection<S>
public abstract List<PCollectionImpl<?>> getParents()
public PCollectionImpl<?> getOnlyParent()
public Set<SourceTarget<?>> getTargetDependencies()
public int getDepth()
public ReadableData<S> asReadable(boolean materialize)
asReadable
in interface PCollection<S>
materialize
- If true, materialize this data before returning a reference to it
protected ReadableData<S> materializedData()
protected abstract ReadableData<S> getReadableDataInternal()
public long getSize()
PCollection
PCollection
in
bytes.
getSize
in interface PCollection<S>
protected abstract long getSizeInternal()
public abstract long getLastModifiedAt()
protected PCollectionImpl<S> getChainingCollection()
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |