Migration Guide: Spark Core
Upgrading from Core 2.4 to 3.0
-
The
org.apache.spark.ExecutorPlugin
interface and related configuration has been replaced withorg.apache.spark.plugin.SparkPlugin
, which adds new functionality. Plugins using the old interface need to be modified to extend the new interfaces. Check the Monitoring guide for more details. -
Deprecated method
TaskContext.isRunningLocally
has been removed. Local execution was removed and it always has returnedfalse
. -
Deprecated method
shuffleBytesWritten
,shuffleWriteTime
andshuffleRecordsWritten
inShuffleWriteMetrics
have been removed. Instead, usebytesWritten
,writeTime
andrecordsWritten
respectively. -
Deprecated method
AccumulableInfo.apply
have been removed because creatingAccumulableInfo
is disallowed. -
Event log file will be written as UTF-8 encoding, and Spark History Server will replay event log files as UTF-8 encoding. Previously Spark writes event log file as default charset of driver JVM process, so Spark History Server of Spark 2.x is needed to read the old event log files in case of incompatible encoding.