Upgrading from PySpark 2.3.0 to 2.3.1 and aboveΒΆ
As of version 2.3.1 Arrow functionality, including
pandas_udf
andtoPandas()
/createDataFrame()
withspark.sql.execution.arrow.enabled
set toTrue
, has been marked as experimental. These are still evolving and not currently recommended for use in production.