org.apache.crunch.lib.join

Class ShardedJoinStrategy<K,U,V>

java.lang.Object

org.apache.crunch.lib.join.ShardedJoinStrategy<K,U,V>

All Implemented Interfaces:

Serializable, JoinStrategy<K,U,V>

public class ShardedJoinStrategy<K,U,V>
extends Object
implements JoinStrategy<K,U,V>

JoinStrategy that splits the key space up into shards.

This strategy is useful when there are multiple values per key on at least one side of the join, and a large proportion of the values are mapped to a small number of keys.

Using this strategy will increase the number of keys being joined, but can increase performance by spreading processing of a single key over multiple reduce groups.

A custom ShardedJoinStrategy.ShardingStrategy can be provided so that only certain keys are sharded, or keys can be sharded in accordance with how many values are mapped to them.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	ShardedJoinStrategy.ShardingStrategy<K> Determines over how many shards a key will be split in a sharded join.

Constructor Summary

Constructors

Constructor and Description
ShardedJoinStrategy(int numShards) Instantiate with a constant number of shards to use for all keys.
ShardedJoinStrategy(int numShards, int numReducers) Instantiate with a constant number of shards to use for all keys.
ShardedJoinStrategy(ShardedJoinStrategy.ShardingStrategy<K> shardingStrategy) Instantiate with a custom sharding strategy.
ShardedJoinStrategy(ShardedJoinStrategy.ShardingStrategy<K> shardingStrategy, int numReducers) Instantiate with a custom sharding strategy and a specified number of reducers.

Method Summary

Modifier and Type	Method and Description
PTable<K,Pair<U,V>>	join(PTable<K,U> left, PTable<K,V> right, JoinType joinType) Join two tables with the given join type.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ShardedJoinStrategy

public ShardedJoinStrategy(int numShards)

Instantiate with a constant number of shards to use for all keys.

Parameters:

numShards - number of shards to use

ShardedJoinStrategy

public ShardedJoinStrategy(int numShards,
                           int numReducers)

Instantiate with a constant number of shards to use for all keys.

Parameters:

numShards - number of shards to use

numReducers - the amount of reducers to run the join with

ShardedJoinStrategy

public ShardedJoinStrategy(ShardedJoinStrategy.ShardingStrategy<K> shardingStrategy)

Instantiate with a custom sharding strategy.

Parameters:

shardingStrategy - strategy to be used for sharding

ShardedJoinStrategy

public ShardedJoinStrategy(ShardedJoinStrategy.ShardingStrategy<K> shardingStrategy,
                           int numReducers)

Instantiate with a custom sharding strategy and a specified number of reducers.

Parameters:

shardingStrategy - strategy to be used for sharding

numReducers - the amount of reducers to run the join with

Method Detail

join

public PTable<K,Pair<U,V>> join(PTable<K,U> left,
                                PTable<K,V> right,
                                JoinType joinType)

Description copied from interface: JoinStrategy

Join two tables with the given join type.

Specified by:

join in interface JoinStrategy<K,U,V>

Parameters:

left - left table to be joined

right - right table to be joined

joinType - type of join to perform

Returns:

joined tables