pyspark.sql.DataFrame.transform

DataFrame.transform(func)

Returns a new DataFrame. Concise syntax for chaining custom transformations.

New in version 3.0.0.

Parameters

funcfunction

a function that takes and returns a DataFrame.

Examples

>>> from pyspark.sql.functions import col
>>> df = spark.createDataFrame([(1, 1.0), (2, 2.0)], ["int", "float"])
>>> def cast_all_to_int(input_df):
...     return input_df.select([col(col_name).cast("int") for col_name in input_df.columns])
>>> def sort_columns_asc(input_df):
...     return input_df.select(*sorted(input_df.columns))
>>> df.transform(cast_all_to_int).transform(sort_columns_asc).show()
+-----+---+
|float|int|
+-----+---+
|    1|  1|
|    2|  2|
+-----+---+

pyspark.sql.DataFrame.toPandas pyspark.sql.DataFrame.union