| gapply {SparkR} | R Documentation |
Groups the SparkDataFrame using the specified columns and applies the R function to each group.
gapply(x, ...) ## S4 method for signature 'GroupedData' gapply(x, func, schema) ## S4 method for signature 'SparkDataFrame' gapply(x, cols, func, schema)
x |
a SparkDataFrame or GroupedData. |
... |
additional argument(s) passed to the method. |
func |
a function to be applied to each group partition specified by grouping column of the SparkDataFrame. See Details. |
schema |
the schema of the resulting SparkDataFrame after the function is applied.
The schema must match to output of |
cols |
grouping columns. |
func is a function of two arguments. The first, usually named key
(though this is not enforced) corresponds to the grouping key, will be an
unnamed list of length(cols) length-one objects corresponding
to the grouping columns' values for the current group.
The second, herein x, will be a local data.frame with the
columns of the input not in cols for the rows corresponding to key.
The output of func must be a data.frame matching schema –
in particular this means the names of the output data.frame are irrelevant
A SparkDataFrame.
gapply(GroupedData) since 2.0.0
gapply(SparkDataFrame) since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run:
##D # Computes the arithmetic mean of the second column by grouping
##D # on the first and third columns. Output the grouping values and the average.
##D
##D df <- createDataFrame (
##D list(list(1L, 1, "1", 0.1), list(1L, 2, "1", 0.2), list(3L, 3, "3", 0.3)),
##D c("a", "b", "c", "d"))
##D
##D # Here our output contains three columns, the key which is a combination of two
##D # columns with data types integer and string and the mean which is a double.
##D schema <- structType(structField("a", "integer"), structField("c", "string"),
##D structField("avg", "double"))
##D result <- gapply(
##D df,
##D c("a", "c"),
##D function(key, x) {
##D # key will either be list(1L, '1') (for the group where a=1L,c='1') or
##D # list(3L, '3') (for the group where a=3L,c='3')
##D y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE)
##D }, schema)
##D
##D # The schema also can be specified in a DDL-formatted string.
##D schema <- "a INT, c STRING, avg DOUBLE"
##D result <- gapply(
##D df,
##D c("a", "c"),
##D function(key, x) {
##D y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE)
##D }, schema)
##D
##D # We can also group the data and afterwards call gapply on GroupedData.
##D # For example:
##D gdf <- group_by(df, "a", "c")
##D result <- gapply(
##D gdf,
##D function(key, x) {
##D y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE)
##D }, schema)
##D collect(result)
##D
##D # Result
##D # ------
##D # a c avg
##D # 3 3 3.0
##D # 1 1 1.5
##D
##D # Fits linear models on iris dataset by grouping on the 'Species' column and
##D # using 'Sepal_Length' as a target variable, 'Sepal_Width', 'Petal_Length'
##D # and 'Petal_Width' as training features.
##D
##D df <- createDataFrame (iris)
##D schema <- structType(structField("(Intercept)", "double"),
##D structField("Sepal_Width", "double"),structField("Petal_Length", "double"),
##D structField("Petal_Width", "double"))
##D df1 <- gapply(
##D df,
##D df$"Species",
##D function(key, x) {
##D m <- suppressWarnings(lm(Sepal_Length ~
##D Sepal_Width + Petal_Length + Petal_Width, x))
##D data.frame(t(coef(m)))
##D }, schema)
##D collect(df1)
##D
##D # Result
##D # ---------
##D # Model (Intercept) Sepal_Width Petal_Length Petal_Width
##D # 1 0.699883 0.3303370 0.9455356 -0.1697527
##D # 2 1.895540 0.3868576 0.9083370 -0.6792238
##D # 3 2.351890 0.6548350 0.2375602 0.2521257
##D
## End(Not run)