org.apache.crunch.lib

Class TopList

java.lang.Object

org.apache.crunch.lib.TopList

public class TopList
extends Object

Tools for creating top lists of items in PTables and PCollections

Constructor Summary

Constructors

Constructor and Description
TopList()

Method Summary

Methods

Modifier and Type	Method and Description
static <X> PTable<X,Long>	globalToplist(PCollection<X> input) Create a list of unique items in the input collection with their count, sorted descending by their frequency.
static <K> PTable<K,Long>	negateCounts(PTable<K,Long> table) When creating toplists, it is often required to sort by count descending.
static <X,Y> PTable<X,Collection<Pair<Long,Y>>>	topNYbyX(PTable<X,Y> input, int n) Create a top-list of elements in the provided PTable, categorised by the key of the input table and using the count of the value part of the input table.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TopList

public TopList()

Method Detail

topNYbyX

public static <X,Y> PTable<X,Collection<Pair<Long,Y>>> topNYbyX(PTable<X,Y> input,
                                                int n)

Create a top-list of elements in the provided PTable, categorised by the key of the input table and using the count of the value part of the input table. Example: if input = Table(Country, Track), then this will give you the most common n tracks for each country.

Type Parameters:

X - group type

Y - value type

Parameters:

input - table of X Y pairs

n - How many Y values to include in the toplist per X (this will be in memory, so don't make this ridiculous)

Returns:

table of each unique X value mapped to a collection of (count, Y) pairs

globalToplist

public static <X> PTable<X,Long> globalToplist(PCollection<X> input)

Create a list of unique items in the input collection with their count, sorted descending by their frequency.

Type Parameters:

X - record type

Parameters:

input - input collection

Returns:

global toplist

negateCounts

public static <K> PTable<K,Long> negateCounts(PTable<K,Long> table)

When creating toplists, it is often required to sort by count descending. As some sort operations don't support order (such as SecondarySort), this method will negate counts so that a natural-ordered sort will produce a descending order.

Type Parameters:

K - key type

Parameters:

table - PTable to process

Returns:

PTable of the same format with the value negated