pyspark.RDD.foreachPartition

RDD.foreachPartition(f: Callable[[Iterable[T]], None]) → None

Applies a function to each partition of this RDD.

New in version 1.0.0.

Parameters

ffunction

a function applied to each partition

See also

RDD.foreach()

pyspark.sql.DataFrame.foreach()

pyspark.sql.DataFrame.foreachPartition()

Examples

>>> def f(iterator):
...     for x in iterator:
...          print(x)
>>> sc.parallelize([1, 2, 3, 4, 5]).foreachPartition(f)

pyspark.RDD.foreach pyspark.RDD.fullOuterJoin