This project has retired. For details please refer to its Attic page.

org.apache.crunch.impl.mr.exec
Class MRExecutor

java.lang.Object
  com.google.common.util.concurrent.AbstractFuture<PipelineResult>
      org.apache.crunch.impl.mr.exec.MRExecutor

All Implemented Interfaces:

com.google.common.util.concurrent.ListenableFuture<PipelineResult>, Future<PipelineResult>, MRPipelineExecution, PipelineExecution

public class MRExecutor
extends com.google.common.util.concurrent.AbstractFuture<PipelineResult>
implements MRPipelineExecution

Provides APIs for job control at runtime to clients. This class has a thread that submits jobs when they become ready, monitors the states of the running jobs, and updates the states of jobs based on the state changes of their depending jobs states. It is thread-safe.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.crunch.PipelineExecution

PipelineExecution.Status

Constructor Summary

MRExecutor(org.apache.hadoop.conf.Configuration conf, Class<?> jarClass, Map<PCollectionImpl<?>,Set<Target>> outputTargets, Map<PCollectionImpl<?>,MaterializableIterable> toMaterialize)

Method Summary

void	addJob(org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob job)
MRPipelineExecution	execute()
PipelineResult	get()
PipelineResult	get(long timeout, TimeUnit unit)
List<MRJob>	getJobs()
String	getPlanDotFile() Returns the .dot file that allows a client to graph the Crunch execution plan for this pipeline.
PipelineResult	getResult() Retrieve the result of a pipeline if it has been completed, otherwise null.
PipelineExecution.Status	getStatus()
protected void	interruptTask()
void	kill() Kills the pipeline if it is running, no-op otherwise.
void	setPlanDotFile(String planDotFile)
void	waitFor(long timeout, TimeUnit timeUnit) Blocks until pipeline completes or the specified waiting time elapsed.
void	waitUntilDone() Blocks until pipeline completes, i.e.

Methods inherited from class com.google.common.util.concurrent.AbstractFuture

addListener, cancel, isCancelled, isDone, set, setException

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.google.common.util.concurrent.ListenableFuture

addListener

Methods inherited from interface java.util.concurrent.Future

cancel, isCancelled, isDone

Constructor Detail

MRExecutor

public MRExecutor(org.apache.hadoop.conf.Configuration conf,
                  Class<?> jarClass,
                  Map<PCollectionImpl<?>,Set<Target>> outputTargets,
                  Map<PCollectionImpl<?>,MaterializableIterable> toMaterialize)

Method Detail

addJob

public void addJob(org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob job)

setPlanDotFile

public void setPlanDotFile(String planDotFile)

execute

public MRPipelineExecution execute()

getPlanDotFile

public String getPlanDotFile()

Description copied from interface: PipelineExecution

Returns the .dot file that allows a client to graph the Crunch execution plan for this pipeline.

Specified by:

getPlanDotFile in interface PipelineExecution

waitFor

public void waitFor(long timeout,
                    TimeUnit timeUnit)
             throws InterruptedException

Description copied from interface: PipelineExecution

Blocks until pipeline completes or the specified waiting time elapsed.

Specified by:

waitFor in interface PipelineExecution

Throws:

InterruptedException

waitUntilDone

public void waitUntilDone()
                   throws InterruptedException

Description copied from interface: PipelineExecution

Blocks until pipeline completes, i.e. SUCCEEDED, FAILED or KILLED.

Specified by:

waitUntilDone in interface PipelineExecution

Throws:

InterruptedException

get

public PipelineResult get()
                   throws InterruptedException,
                          ExecutionException

Specified by:

get in interface Future<PipelineResult>

Overrides:

get in class com.google.common.util.concurrent.AbstractFuture<PipelineResult>

Throws:

InterruptedException

ExecutionException

get

public PipelineResult get(long timeout,
                          TimeUnit unit)
                   throws InterruptedException,
                          TimeoutException,
                          ExecutionException

Specified by:

get in interface Future<PipelineResult>

Overrides:

get in class com.google.common.util.concurrent.AbstractFuture<PipelineResult>

Throws:

InterruptedException

TimeoutException

ExecutionException

getStatus

public PipelineExecution.Status getStatus()

Specified by:

getStatus in interface PipelineExecution

getResult

public PipelineResult getResult()

Description copied from interface: PipelineExecution

Retrieve the result of a pipeline if it has been completed, otherwise null.

Specified by:

getResult in interface PipelineExecution

kill

public void kill()
          throws InterruptedException

Description copied from interface: PipelineExecution

Kills the pipeline if it is running, no-op otherwise. This method only delivers a kill signal to the pipeline, and does not guarantee the pipeline exits on return. To wait for completely exits, use PipelineExecution.waitUntilDone() after this call.

Specified by:

kill in interface PipelineExecution

Throws:

InterruptedException

interruptTask

protected void interruptTask()

Overrides:

interruptTask in class com.google.common.util.concurrent.AbstractFuture<PipelineResult>

getJobs

public List<MRJob> getJobs()

Specified by:

getJobs in interface MRPipelineExecution