Deprecated API

Contents

• Deprecated Interfaces
• Deprecated Classes
• Deprecated Methods
• Deprecated Constructors

Deprecated Interfaces

Interface and Description
org.apache.spark.AccumulableParam use AccumulatorV2. Since 2.0.0.
org.apache.spark.AccumulatorParam use AccumulatorV2. Since 2.0.0.

Deprecated Classes

Class and Description
org.apache.spark.Accumulable use AccumulatorV2. Since 2.0.0.
org.apache.spark.Accumulator use AccumulatorV2. Since 2.0.0.
org.apache.spark.AccumulatorParam.DoubleAccumulatorParam$ use AccumulatorV2. Since 2.0.0.
org.apache.spark.AccumulatorParam.FloatAccumulatorParam$ use AccumulatorV2. Since 2.0.0.
org.apache.spark.AccumulatorParam.IntAccumulatorParam$ use AccumulatorV2. Since 2.0.0.
org.apache.spark.AccumulatorParam.LongAccumulatorParam$ use AccumulatorV2. Since 2.0.0.
org.apache.spark.AccumulatorParam.StringAccumulatorParam$ use AccumulatorV2. Since 2.0.0.
org.apache.spark.sql.hive.HiveContext Use SparkSession.builder.enableHiveSupport instead. Since 2.0.0.
org.apache.spark.ml.feature.OneHotEncoder OneHotEncoderEstimator will be renamed OneHotEncoder and this OneHotEncoder will be removed in 3.0.0.
org.apache.spark.sql.streaming.ProcessingTime use Trigger.ProcessingTime(intervalMs). Since 2.2.0.

Deprecated Methods

Method and Description
org.apache.spark.SparkContext.accumulable(R, AccumulableParam<R, T>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.SparkContext.accumulable(R, String, AccumulableParam<R, T>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulable(T, AccumulableParam<T, R>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulable(T, String, AccumulableParam<T, R>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.SparkContext.accumulableCollection(R, Function1<R, Growable<T>>, ClassTag<R>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulator(double) use sc().doubleAccumulator(). Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulator(double, String) use sc().doubleAccumulator(String). Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulator(int) use sc().longAccumulator(). Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulator(int, String) use sc().longAccumulator(String). Since 2.0.0.
org.apache.spark.SparkContext.accumulator(T, AccumulatorParam<T>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulator(T, AccumulatorParam<T>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.SparkContext.accumulator(T, String, AccumulatorParam<T>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.accumulator(T, String, AccumulatorParam<T>) use AccumulatorV2. Since 2.0.0.
org.apache.spark.sql.streaming.ProcessingTime.apply(Duration) use Trigger.ProcessingTime(interval)
org.apache.spark.scheduler.AccumulableInfo.apply(long, String, Option<String>, String) do not create AccumulableInfo. Since 2.0.0.
org.apache.spark.scheduler.AccumulableInfo.apply(long, String, Option<String>, String, boolean) do not create AccumulableInfo. Since 2.0.0.
org.apache.spark.scheduler.AccumulableInfo.apply(long, String, String) do not create AccumulableInfo. Since 2.0.0.
org.apache.spark.sql.streaming.ProcessingTime.apply(String) use Trigger.ProcessingTime(interval)
org.apache.spark.sql.SQLContext.applySchema(JavaRDD<?>, Class<?>) Use createDataFrame instead. Since 1.3.0.
org.apache.spark.sql.SQLContext.applySchema(JavaRDD<Row>, StructType) Use createDataFrame instead. Since 1.3.0.
org.apache.spark.sql.SQLContext.applySchema(RDD<?>, Class<?>) Use createDataFrame instead. Since 1.3.0.
org.apache.spark.sql.SQLContext.applySchema(RDD<Row>, StructType) Use createDataFrame instead. Since 1.3.0.
org.apache.spark.sql.functions.approxCountDistinct(Column) Use approx_count_distinct. Since 2.1.0.
org.apache.spark.sql.functions.approxCountDistinct(Column, double) Use approx_count_distinct. Since 2.1.0.
org.apache.spark.sql.functions.approxCountDistinct(String) Use approx_count_distinct. Since 2.1.0.
org.apache.spark.sql.functions.approxCountDistinct(String, double) Use approx_count_distinct. Since 2.1.0.
org.apache.spark.scheduler.StageInfo.attemptId() Use attemptNumber instead. Since 2.3.0.
org.apache.spark.sql.SQLContext.clearActive() Use SparkSession.clearActiveSession instead. Since 2.0.0.
org.apache.spark.ml.clustering.KMeansModel.computeCost(Dataset<?>) This method is deprecated and will be removed in 3.0.0. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary.
org.apache.spark.sql.streaming.ProcessingTime.create(long, TimeUnit) use Trigger.ProcessingTime(interval, unit)
org.apache.spark.sql.streaming.ProcessingTime.create(String) use Trigger.ProcessingTime(interval)
org.apache.spark.sql.SQLContext.createExternalTable(String, String) use sparkSession.catalog.createTable instead. Since 2.2.0.
org.apache.spark.sql.catalog.Catalog.createExternalTable(String, String) use createTable instead. Since 2.2.0.
org.apache.spark.sql.SQLContext.createExternalTable(String, String, Map<String, String>) use sparkSession.catalog.createTable instead. Since 2.2.0.
org.apache.spark.sql.SQLContext.createExternalTable(String, String, Map<String, String>) use sparkSession.catalog.createTable instead. Since 2.2.0.
org.apache.spark.sql.catalog.Catalog.createExternalTable(String, String, Map<String, String>) use createTable instead. Since 2.2.0.
org.apache.spark.sql.catalog.Catalog.createExternalTable(String, String, Map<String, String>) use createTable instead. Since 2.2.0.
org.apache.spark.sql.SQLContext.createExternalTable(String, String, String) use sparkSession.catalog.createTable instead. Since 2.2.0.
org.apache.spark.sql.catalog.Catalog.createExternalTable(String, String, String) use createTable instead. Since 2.2.0.
org.apache.spark.sql.SQLContext.createExternalTable(String, String, StructType, Map<String, String>) use sparkSession.catalog.createTable instead. Since 2.2.0.
org.apache.spark.sql.SQLContext.createExternalTable(String, String, StructType, Map<String, String>) use sparkSession.catalog.createTable instead. Since 2.2.0.
org.apache.spark.sql.catalog.Catalog.createExternalTable(String, String, StructType, Map<String, String>) use createTable instead. Since 2.2.0.
org.apache.spark.sql.catalog.Catalog.createExternalTable(String, String, StructType, Map<String, String>) use createTable instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function<Record, T>, Class<T>) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function<Record, T>, Class<T>, String, String) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function<Record, T>, Class<T>, String, String, String, String, String) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, String, String) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function1<Record, T>, ClassTag<T>) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function1<Record, T>, String, String, ClassTag<T>) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function1<Record, T>, String, String, String, String, String, ClassTag<T>) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.streaming.kinesis.KinesisUtils.createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, String, String) Use KinesisInputDStream.builder instead. Since 2.2.0.
org.apache.spark.sql.functions.currentRow() Use Window.currentRow. Since 2.4.0.
org.apache.spark.api.java.JavaSparkContext.doubleAccumulator(double) use sc().doubleAccumulator(). Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.doubleAccumulator(double, String) use sc().doubleAccumulator(String). Since 2.0.0.
org.apache.spark.sql.Dataset.explode(Seq<Column>, Function1<Row, TraversableOnce<A>>, TypeTags.TypeTag<A>) use flatMap() or select() with functions.explode() instead. Since 2.0.0.
org.apache.spark.sql.Dataset.explode(String, String, Function1<A, TraversableOnce<B>>, TypeTags.TypeTag<B>) use flatMap() or select() with functions.explode() instead. Since 2.0.0.
org.apache.spark.mllib.evaluation.MulticlassMetrics.fMeasure() Use accuracy. Since 2.0.0.
org.apache.spark.sql.SQLContext.getOrCreate(SparkContext) Use SparkSession.builder instead. Since 2.0.0.
org.apache.spark.mllib.clustering.KMeans.getRuns() This has no effect and always returns 1. Since 2.1.0.
org.apache.spark.api.java.JavaSparkContext.intAccumulator(int) use sc().longAccumulator(). Since 2.0.0.
org.apache.spark.api.java.JavaSparkContext.intAccumulator(int, String) use sc().longAccumulator(String). Since 2.0.0.
org.apache.spark.TaskContext.isRunningLocally() Local execution was removed, so this always returns false. Since 2.0.0.
org.apache.spark.sql.SQLContext.jdbc(String, String) As of 1.4.0, replaced by read().jdbc().
org.apache.spark.sql.SQLContext.jdbc(String, String, String[]) As of 1.4.0, replaced by read().jdbc().
org.apache.spark.sql.SQLContext.jdbc(String, String, String, long, long, int) As of 1.4.0, replaced by read().jdbc().
org.apache.spark.sql.DataFrameReader.json(JavaRDD<String>) Use json(Dataset[String]) instead. Since 2.2.0.
org.apache.spark.sql.DataFrameReader.json(RDD<String>) Use json(Dataset[String]) instead. Since 2.2.0.
org.apache.spark.sql.SQLContext.jsonFile(String) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonFile(String, double) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonFile(String, StructType) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonRDD(JavaRDD<String>) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonRDD(JavaRDD<String>, double) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonRDD(JavaRDD<String>, StructType) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonRDD(RDD<String>) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonRDD(RDD<String>, double) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.jsonRDD(RDD<String>, StructType) As of 1.4.0, replaced by read().json().
org.apache.spark.sql.SQLContext.load(String) As of 1.4.0, replaced by read().load(path).
org.apache.spark.sql.SQLContext.load(String, Map<String, String>) As of 1.4.0, replaced by read().format(source).options(options).load().
org.apache.spark.sql.SQLContext.load(String, Map<String, String>) As of 1.4.0, replaced by read().format(source).options(options).load().
org.apache.spark.sql.SQLContext.load(String, String) As of 1.4.0, replaced by read().format(source).load(path).
org.apache.spark.sql.SQLContext.load(String, StructType, Map<String, String>) As of 1.4.0, replaced by read().format(source).schema(schema).options(options).load().
org.apache.spark.sql.SQLContext.load(String, StructType, Map<String, String>) As of 1.4.0, replaced by read().format(source).schema(schema).options(options).load().
org.apache.spark.sql.functions.monotonicallyIncreasingId() Use monotonically_increasing_id(). Since 2.0.0.
org.apache.spark.sql.SQLImplicits.newBooleanSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newByteSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newDoubleSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newFloatSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newIntSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newLongSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newProductSeqEncoder(TypeTags.TypeTag<A>) use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newShortSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLImplicits.newStringSeqEncoder() use newSequenceEncoder
org.apache.spark.sql.SQLContext.parquetFile(Seq<String>) Use read.parquet() instead. Since 1.4.0.
org.apache.spark.sql.SQLContext.parquetFile(String...) As of 1.4.0, replaced by read().parquet().
org.apache.spark.mllib.evaluation.MulticlassMetrics.precision() Use accuracy. Since 2.0.0.
org.apache.spark.sql.expressions.Window.rangeBetween(Column, Column) Use the version with Long parameter types. Since 2.4.0.
org.apache.spark.sql.expressions.WindowSpec.rangeBetween(Column, Column) Use the version with Long parameter types. Since 2.4.0.
org.apache.spark.ml.image.ImageSchema.readImages(String) use `spark.read.format("image").load(path)` and this `readImages` will be removed in 3.0.0. Since 2.4.0.
org.apache.spark.ml.image.ImageSchema.readImages(String, SparkSession, boolean, int, boolean, double, long) use `spark.read.format("image").load(path)` and this `readImages` will be removed in 3.0.0. Since 2.4.0.
org.apache.spark.mllib.evaluation.MulticlassMetrics.recall() Use accuracy. Since 2.0.0.
org.apache.spark.sql.Dataset.registerTempTable(String) Use createOrReplaceTempView(viewName) instead. Since 2.0.0.
org.apache.spark.sql.SQLContext.setActive(SQLContext) Use SparkSession.setActiveSession instead. Since 2.0.0.
org.apache.spark.mllib.clustering.KMeans.setRuns(int) This has no effect. Since 2.1.0.
org.apache.spark.sql.functions.toDegrees(Column) Use degrees. Since 2.1.0.
org.apache.spark.sql.functions.toDegrees(String) Use degrees. Since 2.1.0.
org.apache.spark.sql.functions.toRadians(Column) Use radians. Since 2.1.0.
org.apache.spark.sql.functions.toRadians(String) Use radians. Since 2.1.0.
org.apache.spark.mllib.clustering.KMeans.train(RDD<Vector>, int, int, int) Use train method without 'runs'. Since 2.1.0.
org.apache.spark.mllib.clustering.KMeans.train(RDD<Vector>, int, int, int, String) Use train method without 'runs'. Since 2.1.0.
org.apache.spark.mllib.clustering.KMeans.train(RDD<Vector>, int, int, int, String, long) Use train method without 'runs'. Since 2.1.0.
org.apache.spark.sql.functions.unboundedFollowing() Use Window.unboundedFollowing. Since 2.4.0.
org.apache.spark.sql.functions.unboundedPreceding() Use Window.unboundedPreceding. Since 2.4.0.
org.apache.spark.sql.Dataset.unionAll(Dataset<T>) use union(). Since 2.0.0.

Deprecated Constructors

Constructor and Description
org.apache.spark.mllib.regression.LassoWithSGD() Use ml.regression.LinearRegression with elasticNetParam = 1.0. Note the default regParam is 0.01 for LassoWithSGD, but is 0.0 for LinearRegression. Since 2.0.0.
org.apache.spark.mllib.regression.LinearRegressionWithSGD() Use ml.regression.LinearRegression or LBFGS. Since 2.0.0.
org.apache.spark.mllib.classification.LogisticRegressionWithSGD() Use ml.classification.LogisticRegression or LogisticRegressionWithLBFGS. Since 2.0.0.
org.apache.spark.mllib.regression.RidgeRegressionWithSGD() Use ml.regression.LinearRegression with elasticNetParam = 0.0. Note the default regParam is 0.01 for RidgeRegressionWithSGD, but is 0.0 for LinearRegression. Since 2.0.0.
org.apache.spark.sql.SQLContext(JavaSparkContext) Use SparkSession.builder instead. Since 2.0.0.
org.apache.spark.sql.SQLContext(SparkContext) Use SparkSession.builder instead. Since 2.0.0.