| gapplyCollect {SparkR} | R Documentation |
gapplyCollect
Groups the SparkDataFrame using the specified columns, applies the R function to each group and collects the result back to R as data.frame.
gapplyCollect(x, ...) ## S4 method for signature 'GroupedData' gapplyCollect(x, func) ## S4 method for signature 'SparkDataFrame' gapplyCollect(x, cols, func)
x |
a SparkDataFrame or GroupedData. |
... |
additional argument(s) passed to the method. |
func |
a function to be applied to each group partition specified by grouping
column of the SparkDataFrame. The function |
cols |
grouping columns. |
A data.frame.
gapplyCollect(GroupedData) since 2.0.0
gapplyCollect(SparkDataFrame) since 2.0.0
Other SparkDataFrame functions: SparkDataFrame-class,
agg, arrange,
as.data.frame, attach,
cache, coalesce,
collect, colnames,
coltypes,
createOrReplaceTempView,
crossJoin, dapplyCollect,
dapply, describe,
dim, distinct,
dropDuplicates, dropna,
drop, dtypes,
except, explain,
filter, first,
gapply, getNumPartitions,
group_by, head,
histogram, insertInto,
intersect, isLocal,
join, limit,
merge, mutate,
ncol, nrow,
persist, printSchema,
randomSplit, rbind,
registerTempTable, rename,
repartition, sample,
saveAsTable, schema,
selectExpr, select,
showDF, show,
storageLevel, str,
subset, take,
union, unpersist,
withColumn, with,
write.df, write.jdbc,
write.json, write.orc,
write.parquet, write.text
## Not run:
##D Computes the arithmetic mean of the second column by grouping
##D on the first and third columns. Output the grouping values and the average.
##D
##D df <- createDataFrame (
##D list(list(1L, 1, "1", 0.1), list(1L, 2, "1", 0.2), list(3L, 3, "3", 0.3)),
##D c("a", "b", "c", "d"))
##D
##D result <- gapplyCollect(
##D df,
##D c("a", "c"),
##D function(key, x) {
##D y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE)
##D colnames(y) <- c("key_a", "key_c", "mean_b")
##D y
##D })
##D
##D We can also group the data and afterwards call gapply on GroupedData.
##D For Example:
##D gdf <- group_by(df, "a", "c")
##D result <- gapplyCollect(
##D gdf,
##D function(key, x) {
##D y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE)
##D colnames(y) <- c("key_a", "key_c", "mean_b")
##D y
##D })
##D
##D Result
##D ------
##D key_a key_c mean_b
##D 3 3 3.0
##D 1 1 1.5
##D
##D Fits linear models on iris dataset by grouping on the 'Species' column and
##D using 'Sepal_Length' as a target variable, 'Sepal_Width', 'Petal_Length'
##D and 'Petal_Width' as training features.
##D
##D df <- createDataFrame (iris)
##D result <- gapplyCollect(
##D df,
##D df$"Species",
##D function(key, x) {
##D m <- suppressWarnings(lm(Sepal_Length ~
##D Sepal_Width + Petal_Length + Petal_Width, x))
##D data.frame(t(coef(m)))
##D })
##D
##D Result
##D ---------
##D Model X.Intercept. Sepal_Width Petal_Length Petal_Width
##D 1 0.699883 0.3303370 0.9455356 -0.1697527
##D 2 1.895540 0.3868576 0.9083370 -0.6792238
##D 3 2.351890 0.6548350 0.2375602 0.2521257
##D
## End(Not run)