pyspark.RDD.mapPartitionsWithIndex#
- RDD.mapPartitionsWithIndex(f, preservesPartitioning=False)[source]#
Return a new RDD by applying a function to each partition of this RDD, while tracking the index of the original partition.
New in version 0.7.0.
- Parameters
- ffunction
a function to run on each partition of the RDD
- preservesPartitioningbool, optional, default False
indicates whether the input function preserves the partitioner, which should be False unless this is a pair RDD and the input function doesn’t modify the keys
- Returns
See also
Examples
>>> rdd = sc.parallelize([1, 2, 3, 4], 4) >>> def f(splitIndex, iterator): yield splitIndex ... >>> rdd.mapPartitionsWithIndex(f).sum() 6