pyspark.RDDBarrier.mapPartitionsWithIndex#
- RDDBarrier.mapPartitionsWithIndex(f, preservesPartitioning=False)[source]#
Returns a new RDD by applying a function to each partition of the wrapped RDD, while tracking the index of the original partition. And all tasks are launched together in a barrier stage. The interface is the same as
RDD.mapPartitionsWithIndex()
. Please see the API doc there.New in version 3.0.0.
- Parameters
- ffunction
a function to run on each partition of the RDD
- preservesPartitioningbool, optional, default False
indicates whether the input function preserves the partitioner, which should be False unless this is a pair RDD and the input function doesn’t modify the keys
- Returns
See also
Notes
This API is experimental
Examples
>>> rdd = sc.parallelize([1, 2, 3, 4], 4) >>> def f(splitIndex, iterator): yield splitIndex ... >>> barrier = rdd.barrier() >>> barrier <pyspark.core.rdd.RDDBarrier ...> >>> barrier.mapPartitionsWithIndex(f).sum() 6