pyspark.sql.DataFrame.to_pandas_on_spark¶

DataFrame.to_pandas_on_spark(index_col=None)[source]¶

Converts the existing DataFrame into a pandas-on-Spark DataFrame.

If a pandas-on-Spark DataFrame is converted to a Spark DataFrame and then back to pandas-on-Spark, it will lose the index information and the original index will be turned into a normal column.

This is only available if Pandas is installed and available.

Parameters

index_col: str or list of str, optional, default: None: Index column of table in Spark.

See also

pyspark.pandas.frame.DataFrame.to_spark

Examples

>>> df.show()  
+----+----+
|Col1|Col2|
+----+----+
|   a|   1|
|   b|   2|
|   c|   3|
+----+----+

>>> df.to_pandas_on_spark()  
  Col1  Col2
0    a     1
1    b     2
2    c     3

We can specify the index columns.

>>> df.to_pandas_on_spark(index_col="Col1"): 
      Col2
Col1
a        1
b        2
c        3

pyspark.sql.DataFrame.writeTo pyspark.sql.DataFrameNaFunctions.drop