| cov {SparkR} | R Documentation |
Computes a pair-wise frequency table of the given columns. Also known as a contingency table. The number of distinct values for each column should be less than 1e4. At most 1e6 non-zero pair frequencies will be returned.
Calculate the sample covariance of two numerical columns of a DataFrame.
Calculates the correlation of two columns of a DataFrame. Currently only supports the Pearson Correlation Coefficient. For Spearman Correlation, consider using RDD methods found in MLlib's Statistics.
Finding frequent items for columns, possibly with false positives. Using the frequent element count algorithm described in http://dx.doi.org/10.1145/762471.762473, proposed by Karp, Schenker, and Papadimitriou.
Returns a stratified sample without replacement based on the fraction given on each stratum.
cov(x, col1, col2) corr(x, ...) sampleBy(x, col, fractions, seed) ## S4 method for signature 'DataFrame,character,character' crosstab(x, col1, col2) ## S4 method for signature 'DataFrame,character,character' cov(x, col1, col2) ## S4 method for signature 'DataFrame' corr(x, col1, col2, method = "pearson") ## S4 method for signature 'DataFrame,character' freqItems(x, cols, support = 0.01) ## S4 method for signature 'DataFrame,character,list,numeric' sampleBy(x, col, fractions, seed)
x |
A SparkSQL DataFrame |
col1 |
name of the first column. Distinct items will make the first item of each row. |
col2 |
name of the second column. Distinct items will make the column names of the output. |
col |
column that defines strata |
fractions |
A named list giving sampling fraction for each stratum. If a stratum is not specified, we treat its fraction as zero. |
seed |
random seed |
method |
Optional. A character specifying the method for calculating the correlation. only "pearson" is allowed now. |
cols |
A vector column names to search frequent items in. |
support |
(Optional) The minimum frequency for an item to be considered 'frequent'. Should be greater than 1e-4. Default support = 0.01. |
col1 |
the name of the first column |
col2 |
the name of the second column |
x |
A SparkSQL DataFrame |
col1 |
the name of the first column |
col2 |
the name of the second column |
x |
A SparkSQL DataFrame. |
x |
A SparkSQL DataFrame |
a local R data.frame representing the contingency table. The first column of each row will be the distinct values of 'col1' and the column names will be the distinct values of 'col2'. The name of the first column will be '$col1_$col2'. Pairs that have no occurrences will have zero as their counts.
the covariance of the two columns.
The Pearson Correlation Coefficient as a Double.
a local R data.frame with the frequent items in each column
A new DataFrame that represents the stratified sample
## Not run:
##D df <- jsonFile(sqlContext, "/path/to/file.json")
##D ct <- crosstab(df, "title", "gender")
## End(Not run)
## Not run:
##D df <- jsonFile(sqlContext, "/path/to/file.json")
##D cov <- cov(df, "title", "gender")
## End(Not run)
## Not run:
##D df <- jsonFile(sqlContext, "/path/to/file.json")
##D corr <- corr(df, "title", "gender")
##D corr <- corr(df, "title", "gender", method = "pearson")
## End(Not run)
## Not run:
##D df <- jsonFile(sqlContext, "/path/to/file.json")
##D fi = freqItems(df, c("title", "gender"))
## End(Not run)
## Not run:
##D df <- jsonFile(sqlContext, "/path/to/file.json")
##D sample <- sampleBy(df, "key", fractions, 36)
## End(Not run)