This project has retired. For details please refer to its Attic page.
SparkRuntime (Apache Crunch 0.9.0 API)

org.apache.crunch.impl.spark
Class SparkRuntime

java.lang.Object
  extended by com.google.common.util.concurrent.AbstractFuture<PipelineResult>
      extended by org.apache.crunch.impl.spark.SparkRuntime
All Implemented Interfaces:
com.google.common.util.concurrent.ListenableFuture<PipelineResult>, Future<PipelineResult>, PipelineExecution

public class SparkRuntime
extends com.google.common.util.concurrent.AbstractFuture<PipelineResult>
implements PipelineExecution


Nested Class Summary
 
Nested classes/interfaces inherited from interface org.apache.crunch.PipelineExecution
PipelineExecution.Status
 
Constructor Summary
SparkRuntime(SparkPipeline pipeline, org.apache.spark.api.java.JavaSparkContext sparkContext, org.apache.hadoop.conf.Configuration conf, Map<PCollectionImpl<?>,Set<Target>> outputTargets, Map<PCollectionImpl<?>,MaterializableIterable> toMaterialize, Map<PCollection<?>,org.apache.spark.storage.StorageLevel> toCache)
           
 
Method Summary
 SparkRuntime execute()
           
 PipelineResult get()
           
 PipelineResult get(long timeout, TimeUnit unit)
           
 CombineFn getCombineFn()
           
 org.apache.hadoop.conf.Configuration getConfiguration()
           
 String getPlanDotFile()
          Returns the .dot file that allows a client to graph the Crunch execution plan for this pipeline.
 PipelineResult getResult()
          Retrieve the result of a pipeline if it has been completed, otherwise null.
 SparkRuntimeContext getRuntimeContext()
           
 org.apache.spark.api.java.JavaSparkContext getSparkContext()
           
 PipelineExecution.Status getStatus()
           
 org.apache.spark.storage.StorageLevel getStorageLevel(PCollection<?> pcollection)
           
 boolean isValid(org.apache.spark.api.java.JavaRDDLike<?,?> rdd)
           
 void kill()
          Kills the pipeline if it is running, no-op otherwise.
 void setCombineFn(CombineFn combineFn)
           
 void waitFor(long timeout, TimeUnit timeUnit)
          Blocks until pipeline completes or the specified waiting time elapsed.
 void waitUntilDone()
          Blocks until pipeline completes, i.e.
 
Methods inherited from class com.google.common.util.concurrent.AbstractFuture
addListener, cancel, interruptTask, isCancelled, isDone, set, setException
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface com.google.common.util.concurrent.ListenableFuture
addListener
 
Methods inherited from interface java.util.concurrent.Future
cancel, isCancelled, isDone
 

Constructor Detail

SparkRuntime

public SparkRuntime(SparkPipeline pipeline,
                    org.apache.spark.api.java.JavaSparkContext sparkContext,
                    org.apache.hadoop.conf.Configuration conf,
                    Map<PCollectionImpl<?>,Set<Target>> outputTargets,
                    Map<PCollectionImpl<?>,MaterializableIterable> toMaterialize,
                    Map<PCollection<?>,org.apache.spark.storage.StorageLevel> toCache)
Method Detail

setCombineFn

public void setCombineFn(CombineFn combineFn)

getCombineFn

public CombineFn getCombineFn()

execute

public SparkRuntime execute()

getSparkContext

public org.apache.spark.api.java.JavaSparkContext getSparkContext()

getRuntimeContext

public SparkRuntimeContext getRuntimeContext()

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

isValid

public boolean isValid(org.apache.spark.api.java.JavaRDDLike<?,?> rdd)

getStorageLevel

public org.apache.spark.storage.StorageLevel getStorageLevel(PCollection<?> pcollection)

getPlanDotFile

public String getPlanDotFile()
Description copied from interface: PipelineExecution
Returns the .dot file that allows a client to graph the Crunch execution plan for this pipeline.

Specified by:
getPlanDotFile in interface PipelineExecution

waitFor

public void waitFor(long timeout,
                    TimeUnit timeUnit)
             throws InterruptedException
Description copied from interface: PipelineExecution
Blocks until pipeline completes or the specified waiting time elapsed.

Specified by:
waitFor in interface PipelineExecution
Throws:
InterruptedException

waitUntilDone

public void waitUntilDone()
                   throws InterruptedException
Description copied from interface: PipelineExecution
Blocks until pipeline completes, i.e. SUCCEEDED, FAILED or KILLED.

Specified by:
waitUntilDone in interface PipelineExecution
Throws:
InterruptedException

get

public PipelineResult get()
                   throws InterruptedException,
                          ExecutionException
Specified by:
get in interface Future<PipelineResult>
Overrides:
get in class com.google.common.util.concurrent.AbstractFuture<PipelineResult>
Throws:
InterruptedException
ExecutionException

get

public PipelineResult get(long timeout,
                          TimeUnit unit)
                   throws InterruptedException,
                          TimeoutException,
                          ExecutionException
Specified by:
get in interface Future<PipelineResult>
Overrides:
get in class com.google.common.util.concurrent.AbstractFuture<PipelineResult>
Throws:
InterruptedException
TimeoutException
ExecutionException

getStatus

public PipelineExecution.Status getStatus()
Specified by:
getStatus in interface PipelineExecution

getResult

public PipelineResult getResult()
Description copied from interface: PipelineExecution
Retrieve the result of a pipeline if it has been completed, otherwise null.

Specified by:
getResult in interface PipelineExecution

kill

public void kill()
          throws InterruptedException
Description copied from interface: PipelineExecution
Kills the pipeline if it is running, no-op otherwise. This method only delivers a kill signal to the pipeline, and does not guarantee the pipeline exits on return. To wait for completely exits, use PipelineExecution.waitUntilDone() after this call.

Specified by:
kill in interface PipelineExecution
Throws:
InterruptedException


Copyright © 2014 The Apache Software Foundation. All Rights Reserved.