pyspark.RDD.saveAsSequenceFile

RDD.saveAsSequenceFile(path, compressionCodecClass=None)[source]

Output a Python RDD of key-value pairs (of form RDD[(K, V)]) to any Hadoop file system, using the “org.apache.hadoop.io.Writable” types that we convert from the RDD’s key and value types. The mechanism is as follows:

  1. Pyrolite is used to convert pickled Python RDD into RDD of Java objects.

  2. Keys and values of this Java RDD are converted to Writables and written out.

Parameters
pathstr

path to sequence file

compressionCodecClassstr, optional

fully qualified classname of the compression codec class i.e. “org.apache.hadoop.io.compress.GzipCodec” (None by default)

pyspark.RDD.saveAsPickleFile pyspark.RDD.saveAsTextFile