Package | Description |
---|---|
org.apache.crunch |
Client-facing API and core abstractions.
|
org.apache.crunch.fn |
Commonly used functions for manipulating collections.
|
org.apache.crunch.impl.dist | |
org.apache.crunch.impl.dist.collect | |
org.apache.crunch.impl.mem |
In-memory Pipeline implementation for rapid prototyping and testing.
|
org.apache.crunch.impl.spark | |
org.apache.crunch.impl.spark.collect | |
org.apache.crunch.io |
Data input and output for Pipelines.
|
org.apache.crunch.kafka | |
org.apache.crunch.lambda |
Alternative Crunch API using Java 8 features to allow construction of pipelines using lambda functions and method
references.
|
org.apache.crunch.lib |
Joining, sorting, aggregating, and other commonly used functionality.
|
org.apache.crunch.types |
Common functionality for business object serialization.
|
org.apache.crunch.types.avro |
Business object serialization using Apache Avro.
|
org.apache.crunch.types.writable |
Business object serialization using Hadoop's Writables framework.
|
Modifier and Type | Method and Description |
---|---|
PTableType<K,V> |
PTable.getPTableType()
Returns the
PTableType of this PTable . |
PTableType<K,V> |
TableSource.getTableType() |
Modifier and Type | Method and Description |
---|---|
<K,V> PTable<K,V> |
Pipeline.create(Iterable<Pair<K,V>> contents,
PTableType<K,V> ptype)
Creates a
PTable containing the values found in the given Iterable
using an implementation-specific distribution mechanism. |
<K,V> PTable<K,V> |
Pipeline.create(Iterable<Pair<K,V>> contents,
PTableType<K,V> ptype,
CreateOptions options)
Creates a
PTable containing the values found in the given Iterable
using an implementation-specific distribution mechanism. |
<K,V> PTable<K,V> |
Pipeline.emptyPTable(PTableType<K,V> ptype)
Creates an empty
PTable of the given PTable Type . |
<K,V> PTable<K,V> |
PCollection.parallelDo(DoFn<S,Pair<K,V>> doFn,
PTableType<K,V> type)
Similar to the other
parallelDo instance, but returns a
PTable instance instead of a PCollection . |
<K,V> PTable<K,V> |
PCollection.parallelDo(String name,
DoFn<S,Pair<K,V>> doFn,
PTableType<K,V> type)
Similar to the other
parallelDo instance, but returns a
PTable instance instead of a PCollection . |
<K,V> PTable<K,V> |
PCollection.parallelDo(String name,
DoFn<S,Pair<K,V>> doFn,
PTableType<K,V> type,
ParallelDoOptions options)
Similar to the other
parallelDo instance, but returns a
PTable instance instead of a PCollection . |
Modifier and Type | Method and Description |
---|---|
static <K,V> PTableType<V,K> |
SwapFn.tableType(PTableType<K,V> ptt) |
Modifier and Type | Method and Description |
---|---|
static <K,V> PTableType<V,K> |
SwapFn.tableType(PTableType<K,V> ptt) |
Modifier and Type | Method and Description |
---|---|
<K,V> PTable<K,V> |
DistributedPipeline.create(Iterable<Pair<K,V>> contents,
PTableType<K,V> ptype) |
<K,V> PTable<K,V> |
DistributedPipeline.create(Iterable<Pair<K,V>> contents,
PTableType<K,V> ptype,
CreateOptions options) |
<K,V> PTable<K,V> |
DistributedPipeline.emptyPTable(PTableType<K,V> ptype) |
Modifier and Type | Method and Description |
---|---|
PTableType<K,V> |
EmptyPTable.getPTableType() |
PTableType<K,V> |
BaseUnionTable.getPTableType() |
PTableType<K,V> |
BaseInputTable.getPTableType() |
PTableType<K,V> |
BaseDoTable.getPTableType() |
Modifier and Type | Method and Description |
---|---|
<S,K,V> BaseDoTable<K,V> |
PCollectionFactory.createDoTable(String name,
PCollectionImpl<S> chainingCollection,
CombineFn<K,V> combineFn,
DoFn<S,Pair<K,V>> fn,
PTableType<K,V> type) |
<S,K,V> BaseDoTable<K,V> |
PCollectionFactory.createDoTable(String name,
PCollectionImpl<S> chainingCollection,
DoFn<S,Pair<K,V>> fn,
PTableType<K,V> type,
ParallelDoOptions options) |
<K,V> PTable<K,V> |
PCollectionImpl.parallelDo(DoFn<S,Pair<K,V>> fn,
PTableType<K,V> type) |
<K,V> PTable<K,V> |
PCollectionImpl.parallelDo(String name,
DoFn<S,Pair<K,V>> fn,
PTableType<K,V> type) |
<K,V> PTable<K,V> |
PCollectionImpl.parallelDo(String name,
DoFn<S,Pair<K,V>> fn,
PTableType<K,V> type,
ParallelDoOptions options) |
Constructor and Description |
---|
EmptyPTable(DistributedPipeline pipeline,
PTableType<K,V> ptype) |
Modifier and Type | Method and Description |
---|---|
<K,V> PTable<K,V> |
MemPipeline.create(Iterable<Pair<K,V>> contents,
PTableType<K,V> ptype) |
<K,V> PTable<K,V> |
MemPipeline.create(Iterable<Pair<K,V>> contents,
PTableType<K,V> ptype,
CreateOptions options) |
<K,V> PTable<K,V> |
MemPipeline.emptyPTable(PTableType<K,V> ptype) |
static <S,T> PTable<S,T> |
MemPipeline.typedTableOf(PTableType<S,T> ptype,
Iterable<Pair<S,T>> pairs) |
static <S,T> PTable<S,T> |
MemPipeline.typedTableOf(PTableType<S,T> ptype,
S s,
T t,
Object... more) |
Modifier and Type | Method and Description |
---|---|
<K,V> PTable<K,V> |
SparkPipeline.create(Iterable<Pair<K,V>> contents,
PTableType<K,V> ptype,
CreateOptions options) |
<K,V> PTable<K,V> |
SparkPipeline.emptyPTable(PTableType<K,V> ptype) |
Modifier and Type | Method and Description |
---|---|
PTableType<K,V> |
CreatedTable.getPTableType() |
Modifier and Type | Method and Description |
---|---|
<S,K,V> BaseDoTable<K,V> |
SparkCollectFactory.createDoTable(String name,
PCollectionImpl<S> parent,
CombineFn<K,V> combineFn,
DoFn<S,Pair<K,V>> fn,
PTableType<K,V> type) |
<S,K,V> BaseDoTable<K,V> |
SparkCollectFactory.createDoTable(String name,
PCollectionImpl<S> parent,
DoFn<S,Pair<K,V>> fn,
PTableType<K,V> type,
ParallelDoOptions options) |
Constructor and Description |
---|
CreatedTable(SparkPipeline pipeline,
Iterable<Pair<K,V>> contents,
PTableType<K,V> ptype,
CreateOptions options) |
EmptyPTable(DistributedPipeline pipeline,
PTableType<K,V> ptype) |
Modifier and Type | Method and Description |
---|---|
static <K,V> TableSource<K,V> |
From.avroTableFile(List<org.apache.hadoop.fs.Path> paths,
PTableType<K,V> tableType)
Creates a
TableSource<K,V> for reading an Avro key/value file at the given paths. |
static <K,V> TableSource<K,V> |
From.avroTableFile(org.apache.hadoop.fs.Path path,
PTableType<K,V> tableType)
Creates a
TableSource<K,V> for reading an Avro key/value file at the given path. |
Modifier and Type | Method and Description |
---|---|
PTableType<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable> |
KafkaSource.getTableType() |
Modifier and Type | Method and Description |
---|---|
default PTableType<K,V> |
LTable.pType()
Get the underlying
PTableType used to serialize key/value pairs in this table |
Modifier and Type | Method and Description |
---|---|
default <K,V> LTable<K,V> |
LCollection.filterMap(SFunction<S,Optional<Pair<K,V>>> fn,
PTableType<K,V> pType)
Combination of a filter and map operation by using a function with
Optional return type. |
default <K,V> LTable<K,V> |
LCollection.flatMap(SFunction<S,java.util.stream.Stream<Pair<K,V>>> fn,
PTableType<K,V> pType)
Map each element to zero or more output elements using the provided stream-returning function to yield an
LTable |
default <K,V> LTable<K,V> |
LCollection.map(SFunction<S,Pair<K,V>> fn,
PTableType<K,V> pType)
Map the elements of this collection 1-1 through the supplied function to yield an
LTable |
default <K,V> LTable<K,V> |
LCollection.parallelDo(DoFn<S,Pair<K,V>> fn,
PTableType<K,V> pType)
|
default <K,V> LTable<K,V> |
LCollection.parallelDo(LDoFn<S,Pair<K,V>> fn,
PTableType<K,V> pType)
Transform this LCollection using a Lambda-friendly
LDoFn . |
Modifier and Type | Method and Description |
---|---|
static <K,V> Pair<K,V> |
PTables.getDetachedValue(PTableType<K,V> tableType,
Pair<K,V> value)
Create a detached value for a table
Pair . |
static <K,V1,V2,U,V> |
SecondarySort.sortAndApply(PTable<K,Pair<V1,V2>> input,
DoFn<Pair<K,Iterable<Pair<V1,V2>>>,Pair<U,V>> doFn,
PTableType<U,V> ptype)
Perform a secondary sort on the given
PTable instance and then apply a
DoFn to the resulting sorted data to yield an output PTable<U, V> . |
static <K,V1,V2,U,V> |
SecondarySort.sortAndApply(PTable<K,Pair<V1,V2>> input,
DoFn<Pair<K,Iterable<Pair<V1,V2>>>,Pair<U,V>> doFn,
PTableType<U,V> ptype,
int numReducers)
Perform a secondary sort on the given
PTable instance and then apply a
DoFn to the resulting sorted data to yield an output PTable<U, V> , using
the given number of reducers. |
Modifier and Type | Method and Description |
---|---|
PTableType<K,V> |
PGroupedTableType.getTableType() |
<K,V> PTableType<K,V> |
PTypeFamily.tableOf(PType<K> key,
PType<V> value) |
Constructor and Description |
---|
PGroupedTableType(PTableType<K,V> tableType) |
Modifier and Type | Method and Description |
---|---|
<K,V> PTableType<K,V> |
AvroTypeFamily.tableOf(PType<K> key,
PType<V> value) |
Modifier and Type | Method and Description |
---|---|
<K,V> PTableType<K,V> |
WritableTypeFamily.tableOf(PType<K> key,
PType<V> value) |
Copyright © 2017 The Apache Software Foundation. All rights reserved.