Migrating from Koalas to pandas API on Spark#

  • The package name to import should be changed to pyspark.pandas from databricks.koalas.

    # import databricks.koalas as ks
    import pyspark.pandas as ps
    
  • DataFrame.koalas in Koalas DataFrame was renamed to DataFrame.pandas_on_spark in pandas-on-Spark DataFrame. DataFrame.koalas was removed as of Spark 4.0.

  • Monkey-patched DataFrame.to_koalas in PySpark DataFrame was renamed to DataFrame.pandas_api in PySpark DataFrame. DataFrame.to_koalas was removed as of Spark 4.0.

  • Monkey-patched DataFrame.to_pandas_on_spark in PySpark DataFrame was renamed to DataFrame.pandas_api in PySpark DataFrame. DataFrame.to_pandas_on_spark was removed as of Spark 4.0.

  • databricks.koalas.__version__ was removed. pyspark.__version__ should be used instead.