This project has retired. For details please refer to its Attic page.

org.apache.crunch.util
Class DistCache

java.lang.Object
  org.apache.crunch.util.DistCache

public class DistCache
extends Object

Provides functions for working with Hadoop's distributed cache. These include:

• Functions for working with a job-specific distributed cache of objects, like the serialized runtime nodes in a MapReduce.
• Functions for adding library jars to the distributed cache, which will be added to the classpath of MapReduce tasks.

Constructor Summary

DistCache()

Method Summary

static void	addCacheFile(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf)
static void	addJarDirToDistributedCache(org.apache.hadoop.conf.Configuration conf, File jarDirectory) Adds all jars under the specified directory to the distributed cache of jobs using the provided configuration.
static void	addJarDirToDistributedCache(org.apache.hadoop.conf.Configuration conf, String jarDirectory) Adds all jars under the directory at the specified path to the distributed cache of jobs using the provided configuration.
static void	addJarToDistributedCache(org.apache.hadoop.conf.Configuration conf, File jarFile) Adds the specified jar to the distributed cache of jobs using the provided configuration.
static void	addJarToDistributedCache(org.apache.hadoop.conf.Configuration conf, String jarFile) Adds the jar at the specified path to the distributed cache of jobs using the provided configuration.
static String	findContainingJar(Class<?> jarClass) Finds the path to a jar that contains the class provided, if any.
static org.apache.hadoop.fs.Path	getPathToCacheFile(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf)
static Object	read(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path)
static void	write(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path, Object value)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DistCache

public DistCache()

Method Detail

write

public static void write(org.apache.hadoop.conf.Configuration conf,
                         org.apache.hadoop.fs.Path path,
                         Object value)
                  throws IOException

Throws:

IOException

read

public static Object read(org.apache.hadoop.conf.Configuration conf,
                          org.apache.hadoop.fs.Path path)
                   throws IOException

Throws:

IOException

addCacheFile

public static void addCacheFile(org.apache.hadoop.fs.Path path,
                                org.apache.hadoop.conf.Configuration conf)

getPathToCacheFile

public static org.apache.hadoop.fs.Path getPathToCacheFile(org.apache.hadoop.fs.Path path,
                                                           org.apache.hadoop.conf.Configuration conf)

addJarToDistributedCache

public static void addJarToDistributedCache(org.apache.hadoop.conf.Configuration conf,
                                            File jarFile)
                                     throws IOException

Adds the specified jar to the distributed cache of jobs using the provided configuration. The jar will be placed on the classpath of tasks run by the job.

Parameters:

conf - The configuration used to add the jar to the distributed cache.

jarFile - The jar file to add to the distributed cache.

Throws:

IOException - If the jar file does not exist or there is a problem accessing the file.

addJarToDistributedCache

public static void addJarToDistributedCache(org.apache.hadoop.conf.Configuration conf,
                                            String jarFile)
                                     throws IOException

Adds the jar at the specified path to the distributed cache of jobs using the provided configuration. The jar will be placed on the classpath of tasks run by the job.

Parameters:

conf - The configuration used to add the jar to the distributed cache.

jarFile - The path to the jar file to add to the distributed cache.

Throws:

IOException - If the jar file does not exist or there is a problem accessing the file.

findContainingJar

public static String findContainingJar(Class<?> jarClass)
                                throws IOException

Finds the path to a jar that contains the class provided, if any. There is no guarantee that the jar returned will be the first on the classpath to contain the file. This method is basically lifted out of Hadoop's JobConf class.

Parameters:

jarClass - The class the jar file should contain.

Returns:

The path to a jar file that contains the class, or null if no such jar exists.

Throws:

IOException - If there is a problem searching for the jar file.

addJarDirToDistributedCache

public static void addJarDirToDistributedCache(org.apache.hadoop.conf.Configuration conf,
                                               File jarDirectory)
                                        throws IOException

Adds all jars under the specified directory to the distributed cache of jobs using the provided configuration. The jars will be placed on the classpath of tasks run by the job. This method does not descend into subdirectories when adding jars.

Parameters:

conf - The configuration used to add jars to the distributed cache.

jarDirectory - A directory containing jar files to add to the distributed cache.

Throws:

IOException - If the directory does not exist or there is a problem accessing the directory.

addJarDirToDistributedCache

public static void addJarDirToDistributedCache(org.apache.hadoop.conf.Configuration conf,
                                               String jarDirectory)
                                        throws IOException

Adds all jars under the directory at the specified path to the distributed cache of jobs using the provided configuration. The jars will be placed on the classpath of the tasks run by the job. This method does not descend into subdirectories when adding jars.

Parameters:

conf - The configuration used to add jars to the distributed cache.

jarDirectory - The path to a directory containing jar files to add to the distributed cache.

Throws:

IOException - If the directory does not exist or there is a problem accessing the directory.