This project has retired. For details please refer to its Attic page.

org.apache.crunch.types.avro
Class Avros

java.lang.Object
  org.apache.crunch.types.avro.Avros

public class Avros
extends Object

Defines static methods that are analogous to the methods defined in AvroTypeFamily for convenient static importing.

Field Summary

static MapFn<Object,ByteBuffer>	BYTES_IN
static boolean	CAN_COMBINE_SPECIFIC_AND_REFLECT_SCHEMAS Older versions of Avro (i.e., before 1.7.0) do not support schemas that are composed of a mix of specific and reflection-based schemas.
static ReflectDataFactory	REFLECT_DATA_FACTORY Deprecated. as of 0.9.0; use AvroMode.REFLECT.override(ReaderWriterFactory)
static String	REFLECT_DATA_FACTORY_CLASS The name of the configuration parameter that tracks which reflection factory to use.
static MapFn<String,org.apache.avro.util.Utf8>	STRING_TO_UTF8
static MapFn<CharSequence,String>	UTF8_TO_STRING

Method Summary

static AvroType<Boolean>	booleans()
static AvroType<ByteBuffer>	bytes()
static void	checkCombiningSpecificAndReflectionSchemas()
static <T> AvroType<Collection<T>>	collections(PType<T> ptype)
static void	configureReflectDataFactory(org.apache.hadoop.conf.Configuration conf) Deprecated. as of 0.9.0; use AvroMode.REFLECT.configure(Configuration)
static <T> AvroType<T>	containers(Class<T> clazz)
static <S,T> AvroType<T>	derived(Class<T> clazz, MapFn<S,T> inputFn, MapFn<T,S> outputFn, PType<S> base)
static <S,T> AvroType<T>	derivedImmutable(Class<T> clazz, MapFn<S,T> inputFn, MapFn<T,S> outputFn, PType<S> base)
static AvroType<Double>	doubles()
static AvroType<Float>	floats()
static AvroType<org.apache.avro.generic.GenericData.Record>	generics(org.apache.avro.Schema schema)
static <T> PType<T>	getPrimitiveType(Class<T> clazz)
static ReflectDataFactory	getReflectDataFactory(org.apache.hadoop.conf.Configuration conf) Deprecated. as of 0.9.0; use AvroMode.fromConfiguration(conf)
static AvroType<Integer>	ints()
static <T> PType<T>	jsons(Class<T> clazz)
static <K,V> org.apache.crunch.types.avro.AvroKeyValueTableType<K,V>	keyValueTableOf(PType<K> key, PType<V> value) A table type with an Avro type as key and value.
static AvroType<Long>	longs()
static <T> AvroType<Map<String,T>>	maps(PType<T> ptype)
static AvroType<TupleN>	namedTuples(String tupleName, String[] fieldNames, PType[] ptypes)
static <T> org.apache.avro.io.DatumReader<T>	newReader(AvroType<T> type)
static <T> org.apache.avro.io.DatumReader<T>	newReader(org.apache.avro.Schema schema)
static <T> org.apache.avro.io.DatumWriter<T>	newWriter(AvroType<T> type)
static <T> org.apache.avro.io.DatumWriter<T>	newWriter(org.apache.avro.Schema schema)
static AvroType<Void>	nulls()
static <V1,V2> AvroType<Pair<V1,V2>>	pairs(PType<V1> p1, PType<V2> p2)
static <V1,V2,V3,V4> AvroType<Tuple4<V1,V2,V3,V4>>	quads(PType<V1> p1, PType<V2> p2, PType<V3> p3, PType<V4> p4)
static <T> AvroType<T>	records(Class<T> clazz)
static <T> AvroType<T>	reflects(Class<T> clazz)
static <T> AvroType<T>	reflects(Class<T> clazz, org.apache.avro.Schema schema)
static <T> void	register(Class<T> clazz, AvroType<T> ptype)
static <T extends org.apache.avro.specific.SpecificRecord> AvroType<T>	specifics(Class<T> clazz)
static AvroType<String>	strings()
static <K,V> org.apache.crunch.types.avro.AvroTableType<K,V>	tableOf(PType<K> key, PType<V> value) A table type with an Avro type as key and as value.
static <V1,V2,V3> AvroType<Tuple3<V1,V2,V3>>	triples(PType<V1> p1, PType<V2> p2, PType<V3> p3)
static <T extends Tuple> AvroType<T>	tuples(Class<T> clazz, PType... ptypes)
static AvroType<TupleN>	tuples(PType... ptypes)
static PType<Union>	unionOf(PType<?>... ptypes)
static <T extends org.apache.hadoop.io.Writable> AvroType<T>	writables(Class<T> clazz)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CAN_COMBINE_SPECIFIC_AND_REFLECT_SCHEMAS

public static final boolean CAN_COMBINE_SPECIFIC_AND_REFLECT_SCHEMAS

Older versions of Avro (i.e., before 1.7.0) do not support schemas that are composed of a mix of specific and reflection-based schemas. This bit controls whether or not we allow Crunch jobs to be created that involve mixing specific and reflection-based schemas and can be overridden by the client developer.

REFLECT_DATA_FACTORY

public static ReflectDataFactory REFLECT_DATA_FACTORY

Deprecated. as of 0.9.0; use AvroMode.REFLECT.override(ReaderWriterFactory)

The instance we use for generating reflected schemas. In releases up to 0.8.0, this may be modified by clients (e.g., Scrunch.) to override the reader, writer, and data instances used. Configuring the ReaderWriterFactory by setting this field is deprecated. Instead, use AvroMode.override(ReaderWriterFactory).

REFLECT_DATA_FACTORY_CLASS

public static final String REFLECT_DATA_FACTORY_CLASS

The name of the configuration parameter that tracks which reflection factory to use.

See Also:

Constant Field Values

UTF8_TO_STRING

public static MapFn<CharSequence,String> UTF8_TO_STRING

STRING_TO_UTF8

public static MapFn<String,org.apache.avro.util.Utf8> STRING_TO_UTF8

BYTES_IN

public static MapFn<Object,ByteBuffer> BYTES_IN

Method Detail

configureReflectDataFactory

@Deprecated
public static void configureReflectDataFactory(org.apache.hadoop.conf.Configuration conf)

Deprecated. as of 0.9.0; use AvroMode.REFLECT.configure(Configuration)

getReflectDataFactory

public static ReflectDataFactory getReflectDataFactory(org.apache.hadoop.conf.Configuration conf)

Deprecated. as of 0.9.0; use AvroMode.fromConfiguration(conf)

checkCombiningSpecificAndReflectionSchemas

public static void checkCombiningSpecificAndReflectionSchemas()

newReader

public static <T> org.apache.avro.io.DatumReader<T> newReader(org.apache.avro.Schema schema)

newReader

public static <T> org.apache.avro.io.DatumReader<T> newReader(AvroType<T> type)

newWriter

public static <T> org.apache.avro.io.DatumWriter<T> newWriter(org.apache.avro.Schema schema)

newWriter

public static <T> org.apache.avro.io.DatumWriter<T> newWriter(AvroType<T> type)

register

public static <T> void register(Class<T> clazz,
                                AvroType<T> ptype)

getPrimitiveType

public static <T> PType<T> getPrimitiveType(Class<T> clazz)

nulls

public static final AvroType<Void> nulls()

strings

public static final AvroType<String> strings()

longs

public static final AvroType<Long> longs()

ints

public static final AvroType<Integer> ints()

floats

public static final AvroType<Float> floats()

doubles

public static final AvroType<Double> doubles()

booleans

public static final AvroType<Boolean> booleans()

bytes

public static final AvroType<ByteBuffer> bytes()

records

public static final <T> AvroType<T> records(Class<T> clazz)

generics

public static final AvroType<org.apache.avro.generic.GenericData.Record> generics(org.apache.avro.Schema schema)

containers

public static final <T> AvroType<T> containers(Class<T> clazz)

specifics

public static final <T extends org.apache.avro.specific.SpecificRecord> AvroType<T> specifics(Class<T> clazz)

reflects

public static final <T> AvroType<T> reflects(Class<T> clazz)

reflects

public static final <T> AvroType<T> reflects(Class<T> clazz,
                                             org.apache.avro.Schema schema)

writables

public static final <T extends org.apache.hadoop.io.Writable> AvroType<T> writables(Class<T> clazz)

collections

public static final <T> AvroType<Collection<T>> collections(PType<T> ptype)

maps

public static final <T> AvroType<Map<String,T>> maps(PType<T> ptype)

pairs

public static final <V1,V2> AvroType<Pair<V1,V2>> pairs(PType<V1> p1,
                                                        PType<V2> p2)

triples

public static final <V1,V2,V3> AvroType<Tuple3<V1,V2,V3>> triples(PType<V1> p1,
                                                                  PType<V2> p2,
                                                                  PType<V3> p3)

quads

public static final <V1,V2,V3,V4> AvroType<Tuple4<V1,V2,V3,V4>> quads(PType<V1> p1,
                                                                      PType<V2> p2,
                                                                      PType<V3> p3,
                                                                      PType<V4> p4)

tuples

public static final AvroType<TupleN> tuples(PType... ptypes)

namedTuples

public static final AvroType<TupleN> namedTuples(String tupleName,
                                                 String[] fieldNames,
                                                 PType[] ptypes)

tuples

public static <T extends Tuple> AvroType<T> tuples(Class<T> clazz,
                                                   PType... ptypes)

unionOf

public static PType<Union> unionOf(PType<?>... ptypes)

derived

public static final <S,T> AvroType<T> derived(Class<T> clazz,
                                              MapFn<S,T> inputFn,
                                              MapFn<T,S> outputFn,
                                              PType<S> base)

derivedImmutable

public static final <S,T> AvroType<T> derivedImmutable(Class<T> clazz,
                                                       MapFn<S,T> inputFn,
                                                       MapFn<T,S> outputFn,
                                                       PType<S> base)

jsons

public static <T> PType<T> jsons(Class<T> clazz)

tableOf

public static final <K,V> org.apache.crunch.types.avro.AvroTableType<K,V> tableOf(PType<K> key,
                                                                                  PType<V> value)

A table type with an Avro type as key and as value.

The {code PTableType} returned by this method is also compatible with files containing Avro Pairs that are created using the org.apache.avro.mapred.AvroJob class.

Parameters:

key - the PType of the key in the table

value - the PType of the value in the table

Returns:

PTableType for reading and writing avro tables

keyValueTableOf

public static final <K,V> org.apache.crunch.types.avro.AvroKeyValueTableType<K,V> keyValueTableOf(PType<K> key,
                                                                                                  PType<V> value)

A table type with an Avro type as key and value. The PTableType returned by this method is specifically for reading and writing files that are compatible with those created via the org.apache.avro.mapreduce.AvroJob class. For all other Avro table purposes, the tableOf(org.apache.crunch.types.PType, org.apache.crunch.types.PType) method should be used.

Parameters:

key - the PType of the key in the table

value - the PType of the value in the table

Returns:

PTableType for reading and writing files compatible with those created via the org.apache.avro.mapreduce.AvroJob class