org.apache.spark.rdd

Class SequenceFileRDDFunctions<K,V>

Object

org.apache.spark.rdd.SequenceFileRDDFunctions<K,V>

All Implemented Interfaces:

java.io.Serializable, Logging

public class SequenceFileRDDFunctions<K,V>
extends Object
implements Logging, scala.Serializable

Extra functions available on RDDs of (key, value) pairs to create a Hadoop SequenceFile, through an implicit conversion.

See Also:

Serialized Form

Note:

This can't be part of PairRDDFunctions because we need more implicit parameters to convert our keys and values to Writable.

Constructor Summary

Constructors

Constructor and Description
SequenceFileRDDFunctions(RDD<scala.Tuple2<K,V>> self, Class<? extends org.apache.hadoop.io.Writable> _keyWritableClass, Class<? extends org.apache.hadoop.io.Writable> _valueWritableClass, scala.Function1<K,org.apache.hadoop.io.Writable> evidence$1, scala.reflect.ClassTag<K> evidence$2, scala.Function1<V,org.apache.hadoop.io.Writable> evidence$3, scala.reflect.ClassTag<V> evidence$4)

Method Summary

Modifier and Type	Method and Description
void	saveAsSequenceFile(String path, scala.Option<Class<? extends org.apache.hadoop.io.compress.CompressionCodec>> codec) Output the RDD as a Hadoop SequenceFile using the Writable types we infer from the RDD's key and value types.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

initializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

SequenceFileRDDFunctions

public SequenceFileRDDFunctions(RDD<scala.Tuple2<K,V>> self,
                                Class<? extends org.apache.hadoop.io.Writable> _keyWritableClass,
                                Class<? extends org.apache.hadoop.io.Writable> _valueWritableClass,
                                scala.Function1<K,org.apache.hadoop.io.Writable> evidence$1,
                                scala.reflect.ClassTag<K> evidence$2,
                                scala.Function1<V,org.apache.hadoop.io.Writable> evidence$3,
                                scala.reflect.ClassTag<V> evidence$4)

Method Detail

saveAsSequenceFile

public void saveAsSequenceFile(String path,
                               scala.Option<Class<? extends org.apache.hadoop.io.compress.CompressionCodec>> codec)

Output the RDD as a Hadoop SequenceFile using the Writable types we infer from the RDD's key and value types. If the key or value are Writable, then we use their classes directly; otherwise we map primitive types such as Int and Double to IntWritable, DoubleWritable, etc, byte arrays to BytesWritable, and Strings to Text. The path can be on any Hadoop-supported file system.

Parameters:

path - (undocumented)

codec - (undocumented)