dropDuplicates {SparkR} | R Documentation |
Returns a new SparkDataFrame with duplicate rows removed, considering only the subset of columns.
dropDuplicates(x, colNames = columns(x)) ## S4 method for signature 'SparkDataFrame' dropDuplicates(x, colNames = columns(x))
x |
A SparkDataFrame. |
colnames |
A character vector of column names. |
A SparkDataFrame with duplicate rows removed.
Other SparkDataFrame functions: SparkDataFrame-class
,
[[
, agg
,
arrange
, as.data.frame
,
attach
, cache
,
collect
, colnames
,
coltypes
, columns
,
count
, dapply
,
describe
, dim
,
distinct
, dropna
,
drop
, dtypes
,
except
, explain
,
filter
, first
,
group_by
, head
,
histogram
, insertInto
,
intersect
, isLocal
,
join
, limit
,
merge
, mutate
,
ncol
, persist
,
printSchema
,
registerTempTable
, rename
,
repartition
, sample
,
saveAsTable
, selectExpr
,
select
, showDF
,
show
, str
,
take
, unionAll
,
unpersist
, withColumn
,
write.df
, write.jdbc
,
write.json
, write.parquet
,
write.text
## Not run:
##D sc <- sparkR.init()
##D sqlContext <- sparkRSQL.init(sc)
##D path <- "path/to/file.json"
##D df <- read.json(sqlContext, path)
##D dropDuplicates(df)
##D dropDuplicates(df, c("col1", "col2"))
## End(Not run)