pyspark.resource.ResourceProfileBuilder

class pyspark.resource.ResourceProfileBuilder

Resource profile Builder to build a resource profile to associate with an RDD. A ResourceProfile allows the user to specify executor and task requirements for an RDD that will get applied during a stage. This allows the user to change the resource requirements between stages.

New in version 3.1.0.

See also

pyspark.resource.ResourceProfile

Notes

This API is evolving.

Methods

clearExecutorResourceRequests()
clearTaskResourceRequests()
require(resourceRequest)	Add executor resource requests

Attributes

build
executorResources	Returns
taskResources	Returns

pyspark.resource.ResourceProfile pyspark.resource.ExecutorResourceRequest