pyspark.RDD.saveAsNewAPIHadoopFile

RDD.saveAsNewAPIHadoopFile(path: str, outputFormatClass: str, keyClass: Optional[str] = None, valueClass: Optional[str] = None, keyConverter: Optional[str] = None, valueConverter: Optional[str] = None, conf: Optional[Dict[str, str]] = None) → None

Output a Python RDD of key-value pairs (of form RDD[(K, V)]) to any Hadoop file system, using the new Hadoop OutputFormat API (mapreduce package). Key and value types will be inferred if not specified. Keys and values are converted for output using either user specified converters or “org.apache.spark.api.python.JavaToWritableConverter”. The conf is applied on top of the base Hadoop conf associated with the SparkContext of this RDD to create a merged Hadoop MapReduce job configuration for saving the data.

New in version 1.1.0.

Parameters

pathstr

path to Hadoop file

outputFormatClassstr

fully qualified classname of Hadoop OutputFormat (e.g. “org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat”)

keyClassstr, optional

fully qualified classname of key Writable class

(e.g. “org.apache.hadoop.io.IntWritable”, None by default)

valueClassstr, optional

fully qualified classname of value Writable class (e.g. “org.apache.hadoop.io.Text”, None by default)

keyConverterstr, optional

fully qualified classname of key converter (None by default)

valueConverterstr, optional

fully qualified classname of value converter (None by default)

confdict, optional

Hadoop job configuration (None by default)

See also

SparkContext.newAPIHadoopFile()

RDD.saveAsHadoopDataset()

RDD.saveAsNewAPIHadoopDataset()

RDD.saveAsHadoopFile()

RDD.saveAsSequenceFile()

Examples

>>> import os
>>> import tempfile

Set the class of output format

>>> output_format_class = "org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat"

>>> with tempfile.TemporaryDirectory() as d:
...     path = os.path.join(d, "hadoop_file")
...
...     # Write a temporary Hadoop file
...     rdd = sc.parallelize([(1, {3.0: "bb"}), (2, {1.0: "aa"}), (3, {2.0: "dd"})])
...     rdd.saveAsNewAPIHadoopFile(path, output_format_class)
...
...     # Load this Hadoop file as an RDD
...     sorted(sc.sequenceFile(path).collect())
[(1, {3.0: 'bb'}), (2, {1.0: 'aa'}), (3, {2.0: 'dd'})]

pyspark.RDD.saveAsNewAPIHadoopDataset pyspark.RDD.saveAsPickleFile