Function reference • SparkR

Distributed Data Frame

SparkDataFrame-class: S4 class that represents a SparkDataFrame

groupedData(): S4 class that represents a GroupedData

agg() summarize(): summarize

arrange() orderBy(<SparkDataFrame>,<characterOrColumn>): Arrange Rows by Variables

approxQuantile(<SparkDataFrame>,<character>,<numeric>,<numeric>): Calculates the approximate quantiles of numerical columns of a SparkDataFrame

as.data.frame(): Download data from a SparkDataFrame into a R data.frame

attach(<SparkDataFrame>): Attach SparkDataFrame to R search path

broadcast(): broadcast

cache(): Cache

cacheTable(): Cache Table

checkpoint(): checkpoint

collect(): Collects all the elements of a SparkDataFrame and coerces them into an R data.frame.

coltypes() `coltypes<-`(): coltypes

colnames() `colnames<-`() columns() names(<SparkDataFrame>) `names<-`(<SparkDataFrame>): Column Names of SparkDataFrame

count() n(): Count

createDataFrame() as.DataFrame(): Create a SparkDataFrame

createExternalTable(): (Deprecated) Create an external table

createOrReplaceTempView(): Creates a temporary view using the given name.

createTable(): Creates a table based on the dataset in a data source

crossJoin(<SparkDataFrame>,<SparkDataFrame>): CrossJoin

crosstab(<SparkDataFrame>,<character>,<character>): Computes a pair-wise frequency table of the given columns

cube(): cube

describe(): describe

distinct() unique(<SparkDataFrame>): Distinct

dim(<SparkDataFrame>): Returns the dimensions of SparkDataFrame

drop(): drop

dropDuplicates(): dropDuplicates

dropna() na.omit() fillna(): A set of SparkDataFrame functions working with NA values

dtypes(): DataTypes

except(): except

exceptAll(): exceptAll

explain(): Explain

filter() where(): Filter

getNumPartitions(<SparkDataFrame>): getNumPartitions

group_by() groupBy(): GroupBy

head(<SparkDataFrame>): Head

hint(): hint

histogram(<SparkDataFrame>,<characterOrColumn>): Compute histogram statistics for given column

insertInto(): insertInto

intersect(): Intersect

intersectAll(): intersectAll

isLocal(): isLocal

isStreaming(): isStreaming

join(<SparkDataFrame>,<SparkDataFrame>): Join

limit(): Limit

localCheckpoint(): localCheckpoint

merge(): Merges two data frames

mutate() transform(): Mutate

ncol(<SparkDataFrame>): Returns the number of columns in a SparkDataFrame

count(<SparkDataFrame>) nrow(<SparkDataFrame>): Returns the number of rows in a SparkDataFrame

orderBy(): Ordering Columns in a WindowSpec

persist(): Persist

pivot(<GroupedData>,<character>): Pivot a column of the GroupedData and perform the specified aggregation.

printSchema(): Print Schema of a SparkDataFrame

randomSplit(): randomSplit

rbind(): Union two or more SparkDataFrames

rename() withColumnRenamed(): rename

registerTempTable(): (Deprecated) Register Temporary Table

repartition(): Repartition

repartitionByRange(): Repartition by range

rollup(): rollup

sample() sample_frac(): Sample

sampleBy(): Returns a stratified sample without replacement

saveAsTable(): Save the contents of the SparkDataFrame to a data source as a table

schema(): Get schema object

select() `$`(<SparkDataFrame>) `$<-`(<SparkDataFrame>): Select

selectExpr(): SelectExpr

show(<Column>) show(<GroupedData>) show(<SparkDataFrame>) show(<WindowSpec>) show(<StreamingQuery>): show

showDF(): showDF

str(<SparkDataFrame>): Compactly display the structure of a dataset

storageLevel(<SparkDataFrame>): StorageLevel

subset() `[[`(<SparkDataFrame>,<numericOrcharacter>) `[[<-`(<SparkDataFrame>,<numericOrcharacter>) `[`(<SparkDataFrame>): Subset

summary(): summary

take(): Take the first NUM rows of a SparkDataFrame and return the results as a R data.frame

tableToDF(): Create a SparkDataFrame from a SparkSQL table or view

toJSON(<SparkDataFrame>): toJSON

union(): Return a new SparkDataFrame containing the union of rows

unionAll(): Return a new SparkDataFrame containing the union of rows.

unionByName(): Return a new SparkDataFrame containing the union of rows, matched by column names

unpersist(): Unpersist

unpivot() melt(<SparkDataFrame>,<ANY>,<ANY>,<character>,<character>): Unpivot a DataFrame from wide format to long format.

with(): Evaluate a R expression in an environment constructed from a SparkDataFrame

withColumn(): WithColumn

Data import and export

read.df() loadDF(): Load a SparkDataFrame

read.jdbc(): Create a SparkDataFrame representing the database table accessible via JDBC URL

read.json(): Create a SparkDataFrame from a JSON file.

read.orc(): Create a SparkDataFrame from an ORC file.

read.parquet(): Create a SparkDataFrame from a Parquet file.

read.text(): Create a SparkDataFrame from a text file.

write.df() saveDF(): Save the contents of SparkDataFrame to a data source.

write.jdbc(): Save the content of SparkDataFrame to an external database table via JDBC.

write.json(): Save the contents of SparkDataFrame as a JSON file

write.orc(): Save the contents of SparkDataFrame as an ORC file, preserving the schema.

write.parquet(): Save the contents of SparkDataFrame as a Parquet file, preserving the schema.

write.text(): Save the content of SparkDataFrame in a text file at the specified path.

Column functions

approx_count_distinct() approxCountDistinct() collect_list() collect_set() count_distinct() countDistinct() grouping_bit() grouping_id() kurtosis() max_by() min_by() n_distinct() percentile_approx() product() sd() skewness() stddev() std() stddev_pop() stddev_samp() sum_distinct() sumDistinct() var() variance() var_pop() var_samp() max(<Column>) mean(<Column>) min(<Column>) sum(<Column>): Aggregate functions for Column operations

from_avro() to_avro(): Avro processing functions for Column operations

column_collection_functions reverse,Column-method reverse to_json,Column-method to_json to_csv,Column-method to_csv concat,Column-method concat from_json,Column,characterOrstructTypeOrColumn-method from_json schema_of_json,characterOrColumn-method schema_of_json from_csv,Column,characterOrstructTypeOrColumn-method from_csv schema_of_csv,characterOrColumn-method schema_of_csv array_aggregate,characterOrColumn,Column,function-method array_aggregate array_contains,Column-method array_contains array_distinct,Column-method array_distinct array_except,Column,Column-method array_except array_except,Column-method array_exists,characterOrColumn,function-method array_exists array_filter,characterOrColumn,function-method array_filter array_forall,characterOrColumn,function-method array_forall array_intersect,Column,Column-method array_intersect array_intersect,Column-method array_join,Column,character-method array_join array_join,Column-method array_max,Column-method array_max array_min,Column-method array_min array_position,Column-method array_position array_remove,Column-method array_remove array_repeat,Column,numericOrColumn-method array_repeat array_sort,Column-method array_sort array_transform,characterOrColumn,function-method array_transform array_transform,characterOrColumn,characterOrColumn,function-method arrays_overlap,Column,Column-method arrays_overlap arrays_overlap,Column-method array_union,Column,Column-method array_union array_union,Column-method arrays_zip,Column-method arrays_zip arrays_zip_with,characterOrColumn,characterOrColumn,function-method arrays_zip_with shuffle,Column-method shuffle flatten,Column-method flatten map_concat,Column-method map_concat map_entries,Column-method map_entries map_filter,characterOrColumn,function-method map_filter map_from_arrays,Column,Column-method map_from_arrays map_from_arrays,Column-method map_from_entries,Column-method map_from_entries map_keys,Column-method map_keys transform_keys,characterOrColumn,function-method transform_keys transform_values,characterOrColumn,function-method transform_values map_values,Column-method map_values map_zip_with,characterOrColumn,characterOrColumn,function-method map_zip_with element_at,Column-method element_at explode,Column-method explode size,Column-method size slice,Column-method slice sort_array,Column-method sort_array posexplode,Column-method posexplode explode_outer,Column-method explode_outer posexplode_outer,Column-method posexplode_outer: Collection functions for Column operations

add_months() datediff() date_add() date_format() date_sub() from_utc_timestamp() months_between() next_day() to_utc_timestamp(): Date time arithmetic functions for Column operations

bin() bround() cbrt() ceil() conv() cot() csc() hex() hypot() ln() pmod() rint() sec() shiftLeft() shiftleft() shiftRight() shiftright() shiftRightUnsigned() shiftrightunsigned() signum() degrees() toDegrees() radians() toRadians() unhex() width_bucket() abs(<Column>) acos(<Column>) acosh(<Column>) asin(<Column>) asinh(<Column>) atan(<Column>) atanh(<Column>) ceiling(<Column>) cos(<Column>) cosh(<Column>) exp(<Column>) expm1(<Column>) factorial(<Column>) floor(<Column>) log(<Column>) log10(<Column>) log1p(<Column>) log2(<Column>) round(<Column>) sign(<Column>) sin(<Column>) sinh(<Column>) sqrt(<Column>) tan(<Column>) tanh(<Column>) atan2(<Column>): Math functions for Column operations

assert_true() crc32() hash() md5() raise_error() sha1() sha2() xxhash64(): Miscellaneous functions for Column operations

array_to_vector() vector_to_array(): ML functions for Column operations

when() bitwise_not() bitwiseNOT() create_array() create_map() expr() greatest() input_file_name() isnan() least() lit() monotonically_increasing_id() nanvl() negate() negative() positive() rand() randn() spark_partition_id() struct() coalesce(<Column>) is.nan(<Column>) ifelse(<Column>): Non-aggregate functions for Column operations

ascii() base64() bit_length() collate() collation() concat_ws() decode() encode() format_number() format_string() initcap() instr() levenshtein() locate() lower() lpad() ltrim() octet_length() overlay() regexp_extract() regexp_replace() repeat_string() rpad() rtrim() split_string() soundex() substring_index() translate() trim() unbase64() upper() length(<Column>): String functions for Column operations

cume_dist() dense_rank() lag() lead() nth_value() ntile() percent_rank() rank() row_number(): Window functions for Column operations

alias(<Column>) alias(<SparkDataFrame>): alias

asc() asc_nulls_first() asc_nulls_last() contains() desc() desc_nulls_first() desc_nulls_last() getField() getItem() isNaN() isNull() isNotNull() like() rlike() ilike(): A set of operations working with SparkDataFrame columns

avg(): avg

between(): between

cast(): Casts the column to a different data type.

column(): S4 class that represents a SparkDataFrame column

coalesce(): Coalesce

corr(): corr

cov() covar_samp() covar_pop(): cov

dropFields(): dropFields

endsWith(): endsWith

first(): Return the first row of a SparkDataFrame

last(): last

not() `!`(<Column>): !

otherwise(): otherwise

startsWith(): startsWith

substr(<Column>): substr

current_date() current_timestamp() date_trunc() dayofmonth() dayofweek() dayofyear() monthname() dayname() from_unixtime() hour() last_day() make_date() minute() month() quarter() second() timestamp_seconds() to_date() to_timestamp() unix_timestamp() weekofyear() window() year() trunc(<Column>): Date time functions for Column operations

withField(): withField

over(): over

predict(): Makes predictions from a MLlib model

partitionBy(): partitionBy

rangeBetween(): rangeBetween

rowsBetween(): rowsBetween

windowOrderBy(): windowOrderBy

windowPartitionBy(): windowPartitionBy

WindowSpec-class: S4 class that represents a WindowSpec

`%in%`(<Column>): Match a column with given values.

`%<=>%`: %<=>%

Schema Definitions

structField(): structField

structType(): structType

Structured Streaming

StreamingQuery-class: S4 class that represents a StreamingQuery

awaitTermination(): awaitTermination

isActive(): isActive

queryName(): queryName

lastProgress(): lastProgress

read.stream(): Load a streaming SparkDataFrame

status(): status

stopQuery(): stopQuery

withWatermark(): withWatermark

write.stream(): Write the streaming SparkDataFrame to a data source.

Spark MLlib

MLlib is Spark’s machine learning (ML) library

AFTSurvivalRegressionModel-class: S4 class that represents a AFTSurvivalRegressionModel

ALSModel-class: S4 class that represents an ALSModel

BisectingKMeansModel-class: S4 class that represents a BisectingKMeansModel

DecisionTreeClassificationModel-class: S4 class that represents a DecisionTreeClassificationModel

DecisionTreeRegressionModel-class: S4 class that represents a DecisionTreeRegressionModel

FMClassificationModel-class: S4 class that represents a FMClassificationModel

FMRegressionModel-class: S4 class that represents a FMRegressionModel

FPGrowthModel-class: S4 class that represents a FPGrowthModel

GBTClassificationModel-class: S4 class that represents a GBTClassificationModel

GBTRegressionModel-class: S4 class that represents a GBTRegressionModel

GaussianMixtureModel-class: S4 class that represents a GaussianMixtureModel

GeneralizedLinearRegressionModel-class: S4 class that represents a generalized linear model

glm(<formula>,<ANY>,<SparkDataFrame>): Generalized Linear Models (R-compliant)

IsotonicRegressionModel-class: S4 class that represents an IsotonicRegressionModel

KMeansModel-class: S4 class that represents a KMeansModel

KSTest-class: S4 class that represents an KSTest

LDAModel-class: S4 class that represents an LDAModel

LinearRegressionModel-class: S4 class that represents a LinearRegressionModel

LinearSVCModel-class: S4 class that represents an LinearSVCModel

LogisticRegressionModel-class: S4 class that represents an LogisticRegressionModel

MultilayerPerceptronClassificationModel-class: S4 class that represents a MultilayerPerceptronClassificationModel

NaiveBayesModel-class: S4 class that represents a NaiveBayesModel

PowerIterationClustering-class: S4 class that represents a PowerIterationClustering

PrefixSpan-class: S4 class that represents a PrefixSpan

RandomForestClassificationModel-class: S4 class that represents a RandomForestClassificationModel

RandomForestRegressionModel-class: S4 class that represents a RandomForestRegressionModel

fitted(): Get fitted result from a k-means model

freqItems(<SparkDataFrame>,<character>): Finding frequent items for columns, possibly with false positives

spark.als() summary(<ALSModel>) predict(<ALSModel>) write.ml(<ALSModel>,<character>): Alternating Least Squares (ALS) for Collaborative Filtering

spark.bisectingKmeans() summary(<BisectingKMeansModel>) predict(<BisectingKMeansModel>) fitted(<BisectingKMeansModel>) write.ml(<BisectingKMeansModel>,<character>): Bisecting K-Means Clustering Model

spark.decisionTree() summary(<DecisionTreeRegressionModel>) print(<summary.DecisionTreeRegressionModel>) summary(<DecisionTreeClassificationModel>) print(<summary.DecisionTreeClassificationModel>) predict(<DecisionTreeRegressionModel>) predict(<DecisionTreeClassificationModel>) write.ml(<DecisionTreeRegressionModel>,<character>) write.ml(<DecisionTreeClassificationModel>,<character>): Decision Tree Model for Regression and Classification

spark.fmClassifier() summary(<FMClassificationModel>) predict(<FMClassificationModel>) write.ml(<FMClassificationModel>,<character>): Factorization Machines Classification Model

spark.fmRegressor() summary(<FMRegressionModel>) predict(<FMRegressionModel>) write.ml(<FMRegressionModel>,<character>): Factorization Machines Regression Model

spark.fpGrowth() spark.freqItemsets() spark.associationRules() predict(<FPGrowthModel>) write.ml(<FPGrowthModel>,<character>): FP-growth

spark.gaussianMixture() summary(<GaussianMixtureModel>) predict(<GaussianMixtureModel>) write.ml(<GaussianMixtureModel>,<character>): Multivariate Gaussian Mixture Model (GMM)

spark.gbt() summary(<GBTRegressionModel>) print(<summary.GBTRegressionModel>) summary(<GBTClassificationModel>) print(<summary.GBTClassificationModel>) predict(<GBTRegressionModel>) predict(<GBTClassificationModel>) write.ml(<GBTRegressionModel>,<character>) write.ml(<GBTClassificationModel>,<character>): Gradient Boosted Tree Model for Regression and Classification

spark.glm() summary(<GeneralizedLinearRegressionModel>) print(<summary.GeneralizedLinearRegressionModel>) predict(<GeneralizedLinearRegressionModel>) write.ml(<GeneralizedLinearRegressionModel>,<character>): Generalized Linear Models

spark.isoreg() summary(<IsotonicRegressionModel>) predict(<IsotonicRegressionModel>) write.ml(<IsotonicRegressionModel>,<character>): Isotonic Regression Model

spark.kmeans() summary(<KMeansModel>) predict(<KMeansModel>) write.ml(<KMeansModel>,<character>): K-Means Clustering Model

spark.kstest() summary(<KSTest>) print(<summary.KSTest>): (One-Sample) Kolmogorov-Smirnov Test

spark.lda() spark.posterior() spark.perplexity() summary(<LDAModel>) write.ml(<LDAModel>,<character>): Latent Dirichlet Allocation

spark.lm() summary(<LinearRegressionModel>) predict(<LinearRegressionModel>) write.ml(<LinearRegressionModel>,<character>): Linear Regression Model

spark.logit() summary(<LogisticRegressionModel>) predict(<LogisticRegressionModel>) write.ml(<LogisticRegressionModel>,<character>): Logistic Regression Model

spark.mlp() summary(<MultilayerPerceptronClassificationModel>) predict(<MultilayerPerceptronClassificationModel>) write.ml(<MultilayerPerceptronClassificationModel>,<character>): Multilayer Perceptron Classification Model

spark.naiveBayes() summary(<NaiveBayesModel>) predict(<NaiveBayesModel>) write.ml(<NaiveBayesModel>,<character>): Naive Bayes Models

spark.assignClusters(): PowerIterationClustering

spark.findFrequentSequentialPatterns(): PrefixSpan

spark.randomForest() summary(<RandomForestRegressionModel>) print(<summary.RandomForestRegressionModel>) summary(<RandomForestClassificationModel>) print(<summary.RandomForestClassificationModel>) predict(<RandomForestRegressionModel>) predict(<RandomForestClassificationModel>) write.ml(<RandomForestRegressionModel>,<character>) write.ml(<RandomForestClassificationModel>,<character>): Random Forest Model for Regression and Classification

spark.survreg() summary(<AFTSurvivalRegressionModel>) predict(<AFTSurvivalRegressionModel>) write.ml(<AFTSurvivalRegressionModel>,<character>): Accelerated Failure Time (AFT) Survival Regression Model

spark.svmLinear() predict(<LinearSVCModel>) summary(<LinearSVCModel>) write.ml(<LinearSVCModel>,<character>): Linear SVM Model

read.ml(): Load a fitted MLlib model from the input path.

write.ml(): Saves the MLlib model to the input path

Distributed R

dapply dapply,SparkDataFrame,function,characterOrstructType-method: dapply

dapplyCollect dapplyCollect,SparkDataFrame,function-method: dapplyCollect

gapply(): gapply

gapplyCollect(): gapplyCollect

spark.lapply(): Run a function over a list of elements, distributing the computations with Spark

SQL Catalog

currentCatalog(): Returns the current default catalog

currentDatabase(): Returns the current default database

databaseExists(): Checks if the database with the specified name exists.

dropTempTable(): (Deprecated) Drop Temporary Table

dropTempView(): Drops the temporary view with the given view name in the catalog.

functionExists(): Checks if the function with the specified name exists.

getDatabase(): Get the database with the specified name

getFunc(): Get the function with the specified name

getTable(): Get the table with the specified name

listCatalogs(): Returns a list of catalog available

listColumns(): Returns a list of columns for the given table/view in the specified database

listDatabases(): Returns a list of databases available

listFunctions(): Returns a list of functions registered in the specified database

listTables(): Returns a list of tables or views in the specified database

refreshByPath(): Invalidates and refreshes all the cached data and metadata for SparkDataFrame containing path

refreshTable(): Invalidates and refreshes all the cached data and metadata of the given table

recoverPartitions(): Recovers all the partitions in the directory of a table and update the catalog

setCurrentCatalog(): Sets the current default catalog

setCurrentDatabase(): Sets the current default database

tableExists(): Checks if the table with the specified name exists.

tableNames(): Table Names

tables(): Tables

uncacheTable(): Uncache Table

Spark Session and Context

cancelJobGroup(): Cancel active jobs for the specified group

cancelJobsWithTag(): Cancel active jobs that have the specified tag.

clearCache(): Clear Cache

clearJobGroup(): Clear current job group ID and its description

getLocalProperty(): Get a local property set in this thread, or NULL if it is missing. See setLocalProperty.

install.spark(): Download and Install Apache Spark to a Local Directory

setCheckpointDir(): Set checkpoint directory

setJobDescription(): Set a human readable description of the current job.

setInterruptOnCancel(): Set the behavior of job cancellation from jobs started in this thread.

setJobGroup(): Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.

addJobTag(): Add a tag to be assigned to all the jobs started by this thread.

removeJobTag(): Remove a tag previously added to be assigned to all the jobs started by this thread. Noop if such a tag was not added earlier.

getJobTags(): Get the tags that are currently set to be assigned to all the jobs started by this thread.

clearJobTags(): Clear the current thread's job tags.

setLocalProperty(): Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool. To remove/unset property simply set `value` to NULL e.g. setLocalProperty("key", NULL)

setLogLevel(): Set new log level

spark.addFile(): Add a file or directory to be downloaded with this Spark job on every node.

spark.getSparkFiles(): Get the absolute path of a file added through spark.addFile.

spark.getSparkFilesRootDirectory(): Get the root directory that contains files added through spark.addFile.

sparkR.conf(): Get Runtime Config from the current active SparkSession

sparkR.callJMethod(): Call Java Methods

sparkR.callJStatic(): Call Static Java Methods

sparkR.init(): (Deprecated) Initialize a new Spark Context

sparkR.newJObject(): Create Java Objects

sparkR.session(): Get the existing SparkSession or initialize a new SparkSession.

sparkR.session.stop() sparkR.stop(): Stop the Spark Session and Spark Context

sparkR.uiWebUrl(): Get the URL of the SparkUI instance for the current active SparkSession

sparkR.version(): Get version of Spark on which this application is running

sparkRHive.init(): (Deprecated) Initialize a new HiveContext

sparkRSQL.init(): (Deprecated) Initialize a new SQLContext

sql(): SQL Query