This project has retired. For details please refer to its Attic page.

org.apache.crunch.impl.spark
Class SparkRuntimeContext

java.lang.Object
  org.apache.crunch.impl.spark.SparkRuntimeContext

All Implemented Interfaces:

Serializable

public class SparkRuntimeContext
extends Object
implements Serializable

See Also:

Serialized Form

Constructor Summary

SparkRuntimeContext(org.apache.spark.broadcast.Broadcast<org.apache.hadoop.conf.Configuration> broadConf, org.apache.spark.Accumulator<Map<String,Long>> counters)

Method Summary

org.apache.hadoop.conf.Configuration	getConfiguration()
static org.apache.hadoop.mapreduce.TaskInputOutputContext	getTaskIOContext(org.apache.spark.broadcast.Broadcast<org.apache.hadoop.conf.Configuration> conf, org.apache.spark.Accumulator<Map<String,Long>> counters)
void	initialize(DoFn<?,?> fn)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkRuntimeContext

public SparkRuntimeContext(org.apache.spark.broadcast.Broadcast<org.apache.hadoop.conf.Configuration> broadConf,
                           org.apache.spark.Accumulator<Map<String,Long>> counters)

Method Detail

initialize

public void initialize(DoFn<?,?> fn)

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

getTaskIOContext

public static org.apache.hadoop.mapreduce.TaskInputOutputContext getTaskIOContext(org.apache.spark.broadcast.Broadcast<org.apache.hadoop.conf.Configuration> conf,
                                                                                  org.apache.spark.Accumulator<Map<String,Long>> counters)