Deprecated API

Contents

• Deprecated Interfaces
• Deprecated Classes
• Deprecated Methods
• Deprecated Constructors

Deprecated Interfaces

Interface and Description
org.apache.spark.AccumulableParam use AccumulatorV2. Since 2.0.0.
org.apache.spark.AccumulatorParam use AccumulatorV2. Since 2.0.0.

Deprecated Classes

Class and Description
org.apache.spark.Accumulable use AccumulatorV2. Since 2.0.0.
org.apache.spark.Accumulator use AccumulatorV2. Since 2.0.0.
org.apache.spark.AccumulatorParam.DoubleAccumulatorParam$ use AccumulatorV2. Since 2.0.0.
org.apache.spark.AccumulatorParam.FloatAccumulatorParam$ use AccumulatorV2. Since 2.0.0.
org.apache.spark.AccumulatorParam.IntAccumulatorParam$ use AccumulatorV2. Since 2.0.0.
org.apache.spark.AccumulatorParam.LongAccumulatorParam$ use AccumulatorV2. Since 2.0.0.
org.apache.spark.AccumulatorParam.StringAccumulatorParam$ use AccumulatorV2. Since 2.0.0.
org.apache.spark.ui.env.EnvironmentListener This class will be removed in a future release. Since 2.2.0.
org.apache.spark.ui.exec.ExecutorsListener This class will be removed in a future release. Since 2.2.0.
org.apache.spark.sql.hive.HiveContext Use SparkSession.builder.enableHiveSupport instead. Since 2.0.0.
org.apache.spark.ui.jobs.JobProgressListener This class will be removed in a future release. Since 2.2.0.
org.apache.spark.sql.streaming.ProcessingTime use Trigger.ProcessingTime(intervalMs). Since 2.2.0.
org.apache.spark.ui.storage.StorageListener This class will be removed in a future release. Since 2.2.0.
org.apache.spark.storage.StorageStatus This class may be removed or made private in a future release. Since 2.2.0.
org.apache.spark.storage.StorageStatusListener This class will be removed in a future release. Since 2.2.0.

Deprecated Methods

Method and Description
org.apache.spark.SparkContext.accumulable(R, AccumulableParam<R, T>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.SparkContext.accumulable(R, String, AccumulableParam<R, T>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulable(T, AccumulableParam<T, R>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulable(T, String, AccumulableParam<T, R>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.SparkContext.accumulableCollection(R, Function1<R, Growable<T>>, ClassTag<R>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulator(double) use sc().doubleAccumulator(). Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulator(double, String) use sc().doubleAccumulator(String). Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulator(int) use sc().longAccumulator(). Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulator(int, String) use sc().longAccumulator(String). Since 2.0.0.
org.apache.spark.SparkContext.accumulator(T, AccumulatorParam<T>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulator(T, AccumulatorParam<T>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.SparkContext.accumulator(T, String, AccumulatorParam<T>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulator(T, String, AccumulatorParam<T>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.sql.streaming.ProcessingTime.apply(Duration) use Trigger.ProcessingTime(interval)
org.apache.spark.scheduler.AccumulableInfo.apply(long, String, Option<String>, String) do not create AccumulableInfo. Since 2.0.0.
org.apache.spark.scheduler.AccumulableInfo.apply(long, String, Option<String>, String, boolean) do not create AccumulableInfo. Since 2.0.0.
org.apache.spark.scheduler.AccumulableInfo.apply(long, String, String) do not create AccumulableInfo. Since 2.0.0.
org.apache.spark.sql.streaming.ProcessingTime.apply(String) use Trigger.ProcessingTime(interval)
org.apache.spark.sql.SQLContext.applySchema(JavaRDD<?>, Class<?>) Use createDataFrame instead. Since 1.3.0.
org.apache.spark.sql.SQLContext.applySchema(JavaRDD<Row>, StructType) Use createDataFrame instead. Since 1.3.0.
org.apache.spark.sql.SQLContext.applySchema(RDD<?>, Class<?>) Use createDataFrame instead. Since 1.3.0.
org.apache.spark.sql.SQLContext.applySchema(RDD<Row>, StructType) Use createDataFrame instead. Since 1.3.0.
org.apache.spark.sql.functions.approxCountDistinct(Column) Use approx_count_distinct. Since 2.1.0.
org.apache.spark.sql.functions.approxCountDistinct(Column, double) Use approx_count_distinct. Since 2.1.0.
org.apache.spark.sql.functions.approxCountDistinct(String) Use approx_count_distinct. Since 2.1.0.
org.apache.spark.sql.functions.approxCountDistinct(String, double) Use approx_count_distinct. Since 2.1.0.
org.apache.spark.sql.SQLContext.clearActive() Use SparkSession.clearActiveSession instead. Since 2.0.0.
org.apache.spark.sql.streaming.ProcessingTime.create(long, TimeUnit) use Trigger.ProcessingTime(interval, unit)
org.apache.spark.sql.streaming.ProcessingTime.create(String) use Trigger.ProcessingTime(interval)
org.apache.spark.sql.SQLContext.createExternalTable(String, String) use sparkSession.catalog.createTable instead. Since 2.2.0.
org.apache.spark.sql.catalog.Catalog.createExternalTable(String, String) use createTable instead. Since 2.2.0.
org.apache.spark.sql.SQLContext.createExternalTable(String, String, Map<String, String>) use sparkSession.catalog.createTable instead. Since 2.2.0.
org.apache.spark.sql.SQLContext.createExternalTable(String, String, Map<String, String>) use sparkSession.catalog.createTable instead. Since 2.2.0.
org.apache.spark.sql.catalog.Catalog.createExternalTable(String, String, Map<String, String>) use createTable instead. Since 2.2.0.
org.apache.spark.sql.catalog.Catalog.createExternalTable(String, String, Map<String, String>) use createTable instead. Since 2.2.0.
org.apache.spark.sql.SQLContext.createExternalTable(String, String, String) use sparkSession.catalog.createTable instead. Since 2.2.0.
org.apache.spark.sql.catalog.Catalog.createExternalTable(String, String, String) use createTable instead. Since 2.2.0.
org.apache.spark.sql.SQLContext.createExternalTable(String, String, StructType, Map<String, String>) use sparkSession.catalog.createTable instead. Since 2.2.0.
org.apache.spark.sql.SQLContext.createExternalTable(String, String, StructType, Map<String, String>) use sparkSession.catalog.createTable instead. Since 2.2.0.
org.apache.spark.sql.catalog.Catalog.createExternalTable(String, String, StructType, Map<String, String>) use createTable instead. Since 2.2.0.
org.apache.spark.sql.catalog.Catalog.createExternalTable(String, String, StructType, Map<String, String>) use createTable instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function<Record, T>, Class<T>) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function<Record, T>, Class<T>, String, String) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function<Record, T>, Class<T>, String, String, String, String, String) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, String, String) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function1<Record, T>, ClassTag<T>) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function1<Record, T>, String, String, ClassTag<T>) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function1<Record, T>, String, String, String, String, String, ClassTag<T>) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, String, String) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.api.java.JavaSparkContext.doubleAccumulator(double) use sc().doubleAccumulator(). Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.doubleAccumulator(double, String) use sc().doubleAccumulator(String). Since 2.0.0.
org.apache.spark.sql.Dataset.explode(Seq<Column>, Function1<Row, TraversableOnce<A>>, TypeTags.TypeTag<A>) use flatMap() or select() with functions.explode() instead. Since 2.0.0.
org.apache.spark.sql.Dataset.explode(String, String, Function1<A, TraversableOnce<B>>, TypeTags.TypeTag<B>) use flatMap() or select() with functions.explode() instead. Since 2.0.0.
org.apache.spark.mllib.evaluation.MulticlassMetrics.fMeasure() Use accuracy. Since 2.0.0.
org.apache.spark.SparkContext.getExecutorStorageStatus() This method may change or be removed in a future release. Since 2.2.0.
org.apache.spark.sql.SQLContext.getOrCreate(SparkContext) Use SparkSession.builder instead. Since 2.0.0.
org.apache.spark.mllib.clustering.KMeans.getRuns() This has no effect and always returns 1. Since 2.1.0.
org.apache.spark.api.java.JavaSparkContext.intAccumulator(int) use sc().longAccumulator(). Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.intAccumulator(int, String) use sc().longAccumulator(String). Since 2.0.0.
org.apache.spark.TaskContext.isRunningLocally() Local execution was removed, so this always returns false. Since 2.0.0.
org.apache.spark.sql.SQLContext.jdbc(String, String) As of 1.4.0, replaced by read().jdbc().
org.apache.spark.sql.SQLContext.jdbc(String, String, String[]) As of 1.4.0, replaced by read().jdbc().
org.apache.spark.sql.SQLContext.jdbc(String, String, String, long, long, int) As of 1.4.0, replaced by read().jdbc().
org.apache.spark.sql.DataFrameReader.json(JavaRDD<String>) Use json(Dataset[String]) instead. Since 2.2.0.
org.apache.spark.sql.DataFrameReader.json(RDD<String>) Use json(Dataset[String]) instead. Since 2.2.0.
org.apache.spark.sql.SQLContext.jsonFile(String) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonFile(String, double) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonFile(String, StructType) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonRDD(JavaRDD<String>) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonRDD(JavaRDD<String>, double) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonRDD(JavaRDD<String>, StructType) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonRDD(RDD<String>) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonRDD(RDD<String>, double) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonRDD(RDD<String>, StructType) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.load(String) As of 1.4.0, replaced by read().load(path).
org.apache.spark.sql.SQLContext.load(String, Map<String, String>) As of 1.4.0, replaced by read().format(source).options(options).load().
org.apache.spark.sql.SQLContext.load(String, Map<String, String>) As of 1.4.0, replaced by read().format(source).options(options).load().
org.apache.spark.sql.SQLContext.load(String, String) As of 1.4.0, replaced by read().format(source).load(path).
org.apache.spark.sql.SQLContext.load(String, StructType, Map<String, String>) As of 1.4.0, replaced by read().format(source).schema(schema).options(options).load().
org.apache.spark.sql.SQLContext.load(String, StructType, Map<String, String>) As of 1.4.0, replaced by read().format(source).schema(schema).options(options).load().
org.apache.spark.sql.functions.monotonicallyIncreasingId() Use monotonically_increasing_id(). Since 2.0.0.
org.apache.spark.sql.SQLImplicits.newBooleanSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newByteSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newDoubleSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newFloatSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newIntSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newLongSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newProductSeqEncoder(TypeTags.TypeTag<A>) use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newShortSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newStringSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLContext.parquetFile(Seq<String>) Use read.parquet() instead. Since 1.4.0.
org.apache.spark.sql.SQLContext.parquetFile(String...) As of 1.4.0, replaced by read().parquet().
org.apache.spark.mllib.evaluation.MulticlassMetrics.precision() Use accuracy. Since 2.0.0.
org.apache.spark.mllib.evaluation.MulticlassMetrics.recall() Use accuracy. Since 2.0.0.
org.apache.spark.sql.Dataset.registerTempTable(String) Use createOrReplaceTempView(viewName) instead. Since 2.0.0.
org.apache.spark.sql.SQLContext.setActive(SQLContext) Use SparkSession.setActiveSession instead. Since 2.0.0.
org.apache.spark.mllib.clustering.KMeans.setRuns(int) This has no effect. Since 2.1.0.
org.apache.spark.sql.functions.toDegrees(Column) Use degrees. Since 2.1.0.
org.apache.spark.sql.functions.toDegrees(String) Use degrees. Since 2.1.0.
org.apache.spark.sql.functions.toRadians(Column) Use radians. Since 2.1.0.
org.apache.spark.sql.functions.toRadians(String) Use radians. Since 2.1.0.
org.apache.spark.mllib.clustering.KMeans.train(RDD<Vector>, int, int, int) Use train method without 'runs'. Since 2.1.0.
org.apache.spark.mllib.clustering.KMeans.train(RDD<Vector>, int, int, int, String) Use train method without 'runs'. Since 2.1.0.
org.apache.spark.mllib.clustering.KMeans.train(RDD<Vector>, int, int, int, String, long) Use train method without 'runs'. Since 2.1.0.
org.apache.spark.sql.Dataset.unionAll(Dataset<T>) use union(). Since 2.0.0.

Deprecated Constructors

Constructor and Description
org.apache.spark.mllib.regression.LassoWithSGD() Use ml.regression.LinearRegression with elasticNetParam = 1.0. Note the default regParam is 0.01 for LassoWithSGD, but is 0.0 for LinearRegression. Since 2.0.0.
org.apache.spark.mllib.regression.LinearRegressionWithSGD() Use ml.regression.LinearRegression or LBFGS. Since 2.0.0.
org.apache.spark.mllib.classification.LogisticRegressionWithSGD() Use ml.classification.LogisticRegression or LogisticRegressionWithLBFGS. Since 2.0.0.
org.apache.spark.mllib.regression.RidgeRegressionWithSGD() Use ml.regression.LinearRegression with elasticNetParam = 0.0. Note the default regParam is 0.01 for RidgeRegressionWithSGD, but is 0.0 for LinearRegression. Since 2.0.0.
org.apache.spark.sql.SQLContext(JavaSparkContext) Use SparkSession.builder instead. Since 2.0.0.
org.apache.spark.sql.SQLContext(SparkContext) Use SparkSession.builder instead. Since 2.0.0.