pyspark.RDD.withResources¶

RDD.withResources(profile: pyspark.resource.profile.ResourceProfile) → pyspark.rdd.RDD[T][source]¶

Specify a pyspark.resource.ResourceProfile to use when calculating this RDD. This is only supported on certain cluster managers and currently requires dynamic allocation to be enabled. It will result in new executors with the resources specified being acquired to calculate the RDD.

New in version 3.1.0.