PCollection (Apache Crunch 0.14.0 API)

All Known Subinterfaces:

PGroupedTable<K,V>, PTable<K,V>

All Known Implementing Classes:

BaseDoCollection, BaseDoTable, BaseGroupedTable, BaseInputCollection, BaseInputTable, BaseUnionCollection, BaseUnionTable, CreatedCollection, CreatedTable, DoCollection, DoTable, EmptyPCollection, EmptyPCollection, EmptyPTable, EmptyPTable, InputCollection, InputTable, PCollectionImpl, PGroupedTableImpl, PTableBase, UnionCollection, UnionTable
```
public interface PCollection<S>
```
A representation of an immutable, distributed collection of elements that is the fundamental target of computations in Crunch.

Method Summary

All Methods Instance Methods Abstract Methods
Modifier and Type	Method and Description
`PCollection<S>`	`aggregate(Aggregator<S> aggregator)` Returns a `PCollection` that contains the result of aggregating all values in this instance.
`PObject<Collection<S>>`	`asCollection()`
`ReadableData<S>`	`asReadable(boolean materialize)`
`<K> PTable<K,S>`	`by(MapFn<S,K> extractKeyFn, PType<K> keyType)` Apply the given map function to each element of this instance in order to create a `PTable`.
`<K> PTable<K,S>`	`by(String name, MapFn<S,K> extractKeyFn, PType<K> keyType)` Apply the given map function to each element of this instance in order to create a `PTable`.
`PCollection<S>`	`cache()` Marks this data as cached using the default `CachingOptions`.
`PCollection<S>`	`cache(CachingOptions options)` Marks this data as cached using the given `CachingOptions`.
`PTable<S,Long>`	`count()` Returns a `PTable` instance that contains the counts of each unique element of this PCollection.
`PCollection<S>`	`filter(FilterFn<S> filterFn)` Apply the given filter function to this instance and return the resulting `PCollection`.
`PCollection<S>`	`filter(String name, FilterFn<S> filterFn)` Apply the given filter function to this instance and return the resulting `PCollection`.
`PObject<S>`	`first()`
`String`	`getName()` Returns a shorthand name for this PCollection.
`Pipeline`	`getPipeline()` Returns the `Pipeline` associated with this PCollection.
`PType<S>`	`getPType()` Returns the `PType` of this `PCollection`.
`long`	`getSize()` Returns the size of the data represented by this `PCollection` in bytes.
`PTypeFamily`	`getTypeFamily()` Returns the `PTypeFamily` of this `PCollection`.
`PObject<Long>`	`length()` Returns the number of elements represented by this `PCollection`.
`Iterable<S>`	`materialize()` Returns a reference to the data set represented by this PCollection that may be used by the client to read the data locally.
`PObject<S>`	`max()` Returns a `PObject` of the maximum element of this instance.
`PObject<S>`	`min()` Returns a `PObject` of the minimum element of this instance.
`<K,V> PTable<K,V>`	`parallelDo(DoFn<S,Pair<K,V>> doFn, PTableType<K,V> type)` Similar to the other `parallelDo` instance, but returns a `PTable` instance instead of a `PCollection`.
`<T> PCollection<T>`	`parallelDo(DoFn<S,T> doFn, PType<T> type)` Applies the given doFn to the elements of this `PCollection` and returns a new `PCollection` that is the output of this processing.
`<K,V> PTable<K,V>`	`parallelDo(String name, DoFn<S,Pair<K,V>> doFn, PTableType<K,V> type)` Similar to the other `parallelDo` instance, but returns a `PTable` instance instead of a `PCollection`.
`<K,V> PTable<K,V>`	`parallelDo(String name, DoFn<S,Pair<K,V>> doFn, PTableType<K,V> type, ParallelDoOptions options)` Similar to the other `parallelDo` instance, but returns a `PTable` instance instead of a `PCollection`.
`<T> PCollection<T>`	`parallelDo(String name, DoFn<S,T> doFn, PType<T> type)` Applies the given doFn to the elements of this `PCollection` and returns a new `PCollection` that is the output of this processing.
`<T> PCollection<T>`	`parallelDo(String name, DoFn<S,T> doFn, PType<T> type, ParallelDoOptions options)` Applies the given doFn to the elements of this `PCollection` and returns a new `PCollection` that is the output of this processing.
`<Output> Output`	`sequentialDo(String label, PipelineCallable<Output> pipelineCallable)` Adds the materialized data in this `PCollection` as a dependency to the given `PipelineCallable` and registers it with the `Pipeline` associated with this instance.
`PCollection<S>`	`union(PCollection<S>... collections)` Returns a `PCollection` instance that acts as the union of this `PCollection` and the input `PCollection`s.
`PCollection<S>`	`union(PCollection<S> other)` Returns a `PCollection` instance that acts as the union of this `PCollection` and the given `PCollection`.
`PCollection<S>`	`write(Target target)` Write the contents of this `PCollection` to the given `Target`, using the storage format specified by the target.
`PCollection<S>`	`write(Target target, Target.WriteMode writeMode)` Write the contents of this `PCollection` to the given `Target`, using the given `Target.WriteMode` to handle existing targets.

- Method Detail
  - getPipeline
```
Pipeline getPipeline()
```
    Returns the Pipeline associated with this PCollection.
  - union
```
PCollection<S> union(PCollection<S> other)
```
    Returns a PCollection instance that acts as the union of this PCollection and the given PCollection.
  - union
```
PCollection<S> union(PCollection<S>... collections)
```
    Returns a PCollection instance that acts as the union of this PCollection and the input PCollections.
  - parallelDo
```
<T> PCollection<T> parallelDo(DoFn<S,T> doFn,
                              PType<T> type)
```
    Applies the given doFn to the elements of this PCollection and returns a new PCollection that is the output of this processing.
    
    Parameters:
    
    doFn - The DoFn to apply
    
    type - The PType of the resulting PCollection
    
    Returns:
    
    a new PCollection
  - parallelDo
```
<T> PCollection<T> parallelDo(String name,
                              DoFn<S,T> doFn,
                              PType<T> type)
```
    Applies the given doFn to the elements of this PCollection and returns a new PCollection that is the output of this processing.
    
    Parameters:
    
    name - An identifier for this processing step, useful for debugging
    
    doFn - The DoFn to apply
    
    type - The PType of the resulting PCollection
    
    Returns:
    
    a new PCollection
  - parallelDo
```
<T> PCollection<T> parallelDo(String name,
                              DoFn<S,T> doFn,
                              PType<T> type,
                              ParallelDoOptions options)
```
    Applies the given doFn to the elements of this PCollection and returns a new PCollection that is the output of this processing.
    
    Parameters:
    
    name - An identifier for this processing step, useful for debugging
    
    doFn - The DoFn to apply
    
    type - The PType of the resulting PCollection
    
    options - Optional information that is needed for certain pipeline operations
    
    Returns:
    
    a new PCollection
  - parallelDo
```
<K,V> PTable<K,V> parallelDo(DoFn<S,Pair<K,V>> doFn,
                             PTableType<K,V> type)
```
    Similar to the other parallelDo instance, but returns a PTable instance instead of a PCollection.
    
    Parameters:
    
    doFn - The DoFn to apply
    
    type - The PTableType of the resulting PTable
    
    Returns:
    
    a new PTable
  - parallelDo
```
<K,V> PTable<K,V> parallelDo(String name,
                             DoFn<S,Pair<K,V>> doFn,
                             PTableType<K,V> type)
```
    Similar to the other parallelDo instance, but returns a PTable instance instead of a PCollection.
    
    Parameters:
    
    name - An identifier for this processing step
    
    doFn - The DoFn to apply
    
    type - The PTableType of the resulting PTable
    
    Returns:
    
    a new PTable
  - parallelDo
```
<K,V> PTable<K,V> parallelDo(String name,
                             DoFn<S,Pair<K,V>> doFn,
                             PTableType<K,V> type,
                             ParallelDoOptions options)
```
    Similar to the other parallelDo instance, but returns a PTable instance instead of a PCollection.
    
    Parameters:
    
    name - An identifier for this processing step
    
    doFn - The DoFn to apply
    
    type - The PTableType of the resulting PTable
    
    options - Optional information that is needed for certain pipeline operations
    
    Returns:
    
    a new PTable
  - write
```
PCollection<S> write(Target target)
```
    Write the contents of this PCollection to the given Target, using the storage format specified by the target.
    
    Parameters:
    
    target - The target to write to
  - write
```
PCollection<S> write(Target target,
                     Target.WriteMode writeMode)
```
    Write the contents of this PCollection to the given Target, using the given Target.WriteMode to handle existing targets.
    
    Parameters:
    
    target - The target
    
    writeMode - The rule for handling existing outputs at the target location
  - materialize
```
Iterable<S> materialize()
```
    Returns a reference to the data set represented by this PCollection that may be used by the client to read the data locally.
  - cache
```
PCollection<S> cache()
```
    Marks this data as cached using the default CachingOptions. Cached PCollections will only be processed once, and then their contents will be saved so that downstream code can process them many times.
    
    Returns:
    
    this PCollection instance
  - cache
```
PCollection<S> cache(CachingOptions options)
```
    Marks this data as cached using the given CachingOptions. Cached PCollections will only be processed once and then their contents will be saved so that downstream code can process them many times.
    
    Parameters:
    
    options - the options that control the cache settings for the data
    
    Returns:
    
    this PCollection instance
  - asCollection
```
PObject<Collection<S>> asCollection()
```
    Returns:
    
    A PObject encapsulating an in-memory Collection containing the values of this PCollection.
  - first
```
PObject<S> first()
```
    Returns:
    
    The first element of this PCollection.
  - sequentialDo
```
<Output> Output sequentialDo(String label,
                             PipelineCallable<Output> pipelineCallable)
```
    Adds the materialized data in this PCollection as a dependency to the given PipelineCallable and registers it with the Pipeline associated with this instance.
    
    Parameters:
    
    label - the label to use inside of the PipelineCallable for referencing this PCollection
    
    pipelineCallable - the function itself
    
    Returns:
    
    The value of the getOutput function on the given argument.
  - asReadable
```
ReadableData<S> asReadable(boolean materialize)
```
    Parameters:
    
    materialize - If true, materialize this data before returning a reference to it
    
    Returns:
    
    A reference to the data in this instance that can be read from a job running on a cluster.
  - getPType
```
PType<S> getPType()
```
    Returns the PType of this PCollection.
  - getTypeFamily
```
PTypeFamily getTypeFamily()
```
    Returns the PTypeFamily of this PCollection.
  - getSize
```
long getSize()
```
    Returns the size of the data represented by this PCollection in bytes.
  - length
```
PObject<Long> length()
```
    Returns the number of elements represented by this PCollection.
    
    Returns:
    
    An PObject containing the number of elements in this PCollection.
  - getName
```
String getName()
```
    Returns a shorthand name for this PCollection.
  - filter
```
PCollection<S> filter(FilterFn<S> filterFn)
```
    Apply the given filter function to this instance and return the resulting PCollection.
  - filter
```
PCollection<S> filter(String name,
                      FilterFn<S> filterFn)
```
    Apply the given filter function to this instance and return the resulting PCollection.
    
    Parameters:
    
    name - An identifier for this processing step
    
    filterFn - The FilterFn to apply
  - by
```
<K> PTable<K,S> by(MapFn<S,K> extractKeyFn,
                   PType<K> keyType)
```
    Apply the given map function to each element of this instance in order to create a PTable.
  - by
```
<K> PTable<K,S> by(String name,
                   MapFn<S,K> extractKeyFn,
                   PType<K> keyType)
```
    Apply the given map function to each element of this instance in order to create a PTable.
    
    Parameters:
    
    name - An identifier for this processing step
    
    extractKeyFn - The MapFn to apply
  - count
```
PTable<S,Long> count()
```
    Returns a PTable instance that contains the counts of each unique element of this PCollection.
  - max
```
PObject<S> max()
```
    Returns a PObject of the maximum element of this instance.
  - min
```
PObject<S> min()
```
    Returns a PObject of the minimum element of this instance.
  - aggregate
```
PCollection<S> aggregate(Aggregator<S> aggregator)
```
    Returns a PCollection that contains the result of aggregating all values in this instance.

Interface PCollection<S>

Method Summary

Method Detail

getPipeline

union

union

parallelDo

parallelDo

parallelDo

parallelDo

parallelDo

parallelDo

write

write

materialize

cache

cache

asCollection

first

sequentialDo

asReadable

getPType

getTypeFamily

getSize

length

getName

filter

filter

by

by

count

max

min

aggregate