org.apache.crunch.util

Class DistCache

java.lang.Object

org.apache.crunch.util.DistCache

public class DistCache
extends Object

Provides functions for working with Hadoop's distributed cache. These include:

• Functions for working with a job-specific distributed cache of objects, like the serialized runtime nodes in a MapReduce.
• Functions for adding library jars to the distributed cache, which will be added to the classpath of MapReduce tasks.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DIST_CACHE_REPLICATION Configuration key for setting the replication factor for files distributed using the Crunch DistCache helper class.

Constructor Summary

Constructors

Constructor and Description
DistCache()

Method Summary

Modifier and Type	Method and Description
static void	addCacheFile(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf)
static void	addJarDirToDistributedCache(org.apache.hadoop.conf.Configuration conf, File jarDirectory) Adds all jars under the specified directory to the distributed cache of jobs using the provided configuration.
static void	addJarDirToDistributedCache(org.apache.hadoop.conf.Configuration conf, String jarDirectory) Adds all jars under the directory at the specified path to the distributed cache of jobs using the provided configuration.
static void	addJarToDistributedCache(org.apache.hadoop.conf.Configuration conf, File jarFile) Adds the specified jar to the distributed cache of jobs using the provided configuration.
static void	addJarToDistributedCache(org.apache.hadoop.conf.Configuration conf, String jarFile) Adds the jar at the specified path to the distributed cache of jobs using the provided configuration.
static String	findContainingJar(Class<?> jarClass) Finds the path to a jar that contains the class provided, if any.
static org.apache.hadoop.fs.Path	getPathToCacheFile(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf)
static Object	read(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path)
static void	write(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path, Object value)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DIST_CACHE_REPLICATION

public static final String DIST_CACHE_REPLICATION

Configuration key for setting the replication factor for files distributed using the Crunch DistCache helper class. This can be used to scale read access for files used by the Crunch framework.

See Also:

Constant Field Values

Constructor Detail

DistCache

public DistCache()

Method Detail

write

public static void write(org.apache.hadoop.conf.Configuration conf,
                         org.apache.hadoop.fs.Path path,
                         Object value)
                  throws IOException

Throws:

IOException

read

public static Object read(org.apache.hadoop.conf.Configuration conf,
                          org.apache.hadoop.fs.Path path)
                   throws IOException

Throws:

IOException

addCacheFile

public static void addCacheFile(org.apache.hadoop.fs.Path path,
                                org.apache.hadoop.conf.Configuration conf)

getPathToCacheFile

public static org.apache.hadoop.fs.Path getPathToCacheFile(org.apache.hadoop.fs.Path path,
                                                           org.apache.hadoop.conf.Configuration conf)

addJarToDistributedCache

public static void addJarToDistributedCache(org.apache.hadoop.conf.Configuration conf,
                                            File jarFile)
                                     throws IOException

Adds the specified jar to the distributed cache of jobs using the provided configuration. The jar will be placed on the classpath of tasks run by the job.

Parameters:

conf - The configuration used to add the jar to the distributed cache.

jarFile - The jar file to add to the distributed cache.

Throws:

IOException - If the jar file does not exist or there is a problem accessing the file.

addJarToDistributedCache

public static void addJarToDistributedCache(org.apache.hadoop.conf.Configuration conf,
                                            String jarFile)
                                     throws IOException

Adds the jar at the specified path to the distributed cache of jobs using the provided configuration. The jar will be placed on the classpath of tasks run by the job.

Parameters:

conf - The configuration used to add the jar to the distributed cache.

jarFile - The path to the jar file to add to the distributed cache.

Throws:

IOException - If the jar file does not exist or there is a problem accessing the file.

findContainingJar

public static String findContainingJar(Class<?> jarClass)
                                throws IOException

Finds the path to a jar that contains the class provided, if any. There is no guarantee that the jar returned will be the first on the classpath to contain the file. This method is basically lifted out of Hadoop's JobConf class.

Parameters:

jarClass - The class the jar file should contain.

Returns:

The path to a jar file that contains the class, or null if no such jar exists.

Throws:

IOException - If there is a problem searching for the jar file.

addJarDirToDistributedCache

public static void addJarDirToDistributedCache(org.apache.hadoop.conf.Configuration conf,
                                               File jarDirectory)
                                        throws IOException

Adds all jars under the specified directory to the distributed cache of jobs using the provided configuration. The jars will be placed on the classpath of tasks run by the job. This method does not descend into subdirectories when adding jars.

Parameters:

conf - The configuration used to add jars to the distributed cache.

jarDirectory - A directory containing jar files to add to the distributed cache.

Throws:

IOException - If the directory does not exist or there is a problem accessing the directory.

addJarDirToDistributedCache

public static void addJarDirToDistributedCache(org.apache.hadoop.conf.Configuration conf,
                                               String jarDirectory)
                                        throws IOException

Adds all jars under the directory at the specified path to the distributed cache of jobs using the provided configuration. The jars will be placed on the classpath of the tasks run by the job. This method does not descend into subdirectories when adding jars.

Parameters:

conf - The configuration used to add jars to the distributed cache.

jarDirectory - The path to a directory containing jar files to add to the distributed cache.

Throws:

IOException - If the directory does not exist or there is a problem accessing the directory.