pyspark.RDDBarrier.mapPartitions

RDDBarrier.mapPartitions(f, preservesPartitioning=False)[source]

Returns a new RDD by applying a function to each partition of the wrapped RDD, where tasks are launched together in a barrier stage. The interface is the same as RDD.mapPartitions(). Please see the API doc there.

New in version 2.4.0.

Notes

This API is experimental

pyspark.TaskContext.taskAttemptId pyspark.RDDBarrier.mapPartitionsWithIndex