org.apache.crunch.lib

Class Quantiles

java.lang.Object

org.apache.crunch.lib.Quantiles

public class Quantiles
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Quantiles.Result<V> Output type for storing the results of a Quantiles computation

Constructor Summary

Constructors

Constructor and Description
Quantiles()

Method Summary

Modifier and Type	Method and Description
static <K,V extends Number> PTable<K,Quantiles.Result<V>>	distributed(PTable<K,V> table, double p1, double... pn) Calculate a set of quantiles for each key in a numerically-valued table.
static <K,V extends Comparable> PTable<K,Quantiles.Result<V>>	inMemory(PTable<K,V> table, double p1, double... pn) Calculate a set of quantiles for each key in a numerically-valued table.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Quantiles

public Quantiles()

Method Detail

distributed

public static <K,V extends Number> PTable<K,Quantiles.Result<V>> distributed(PTable<K,V> table,
                                                                             double p1,
                                                                             double... pn)

Calculate a set of quantiles for each key in a numerically-valued table. Quantiles are calculated on a per-key basis by counting, joining and sorting. This is highly scalable, but takes 2 more map-reduce cycles than if you can guarantee that the value set will fit into memory. Use inMemory if you have less than the order of 10M values per key. The quantile definition that we use here is the "nearest rank" defined here: http://en.wikipedia.org/wiki/Percentile#Definition

Type Parameters:

K - Key type of the table

V - Value type of the table (must extends java.lang.Number)

Parameters:

table - numerically-valued PTable

p1 - First quantile (in the range 0.0 - 1.0)

pn - More quantiles (in the range 0.0 - 1.0)

Returns:

PTable of each key with a collection of pairs of the quantile provided and it's result.

inMemory

public static <K,V extends Comparable> PTable<K,Quantiles.Result<V>> inMemory(PTable<K,V> table,
                                                                              double p1,
                                                                              double... pn)

Calculate a set of quantiles for each key in a numerically-valued table. Quantiles are calculated on a per-key basis by grouping, reading the data into memory, then sorting and and calculating. This is much faster than the distributed option, but if you get into the order of 10M+ per key, then performance might start to degrade or even cause OOMs. The quantile definition that we use here is the "nearest rank" defined here: http://en.wikipedia.org/wiki/Percentile#Definition

Type Parameters:

K - Key type of the table

V - Value type of the table (must extends java.lang.Number)

Parameters:

table - numerically-valued PTable

p1 - First quantile (in the range 0.0 - 1.0)

pn - More quantiles (in the range 0.0 - 1.0)

Returns:

PTable of each key with a collection of pairs of the quantile provided and it's result.