Spark 2.4.8 ScalaDoc

ComplexFutureAction FutureAction SimpleFutureAction

canDoMerge

HiveUDAFBuffer

canEqual

ExecutorInfo MutablePair

canHandle

JdbcDialect

canWrite

DataType

cancel

cancelAllJobs

cancelJob

SparkContext

cancelJobGroup

SparkSession sql HiveSessionStateBuilder

cancelStage

cartesian

caseSensitive

cast

catalog

catalogString

ArrayType DataType MapType StructType

categoricalCols

FeatureHasher

categoricalFeaturesInfo

categories

categoryMaps

categorySizes

cause

AnalysisException StreamingQueryException

cbrt

JavaRDDLike Graph EdgeRDDImpl GraphImpl VertexRDDImpl HadoopRDD RDD Dataset StreamingContext JavaDStreamLike JavaStreamingContext DStream

ceil

functions Decimal

changePrecision

Decimal

checkSingleVsMultiColumnParams

ParamValidators

checkpoint

checkpointAppName

Builder

checkpointFile

HasCheckpointInterval Strategy Builder

checkpointInterval

chiSqTest

Statistics

child

ScriptTransformationExec Not

className

ExceptionFailure Function

classTag

JavaDoubleRDD JavaPairRDD JavaRDD JavaRDDLike JavaDStream JavaDStreamLike JavaInputDStream JavaPairDStream JavaReceiverInputDStream

classification

ml mllib

classpathEntries

ApplicationEnvironmentInfo

clean

WriteAheadLog

clear

Params ExecutionListenerManager

~~clearActive~~

~~SQLContext~~

clearActiveSession

SparkSession

clearCache

SQLContext Catalog

clearCallSite

clearDefaultSession

SparkSession

clearDependencies

CoGroupedRDD RDD ShuffledRDD UnionRDD

clearJobGroup

LogisticRegressionModel SVMModel

clearThreshold

clone

SparkConf ExperimentalMethods Decimal ExecutionListenerManager StorageLevel BernoulliCellSampler BernoulliSampler PoissonSampler RandomSampler

cloneComplement

BernoulliCellSampler

close

JavaSparkContext NioBufferedFileInputStream ReadAheadInputStream DeserializationStream SerializationStream ForeachWriter SparkSession HiveOutputWriter ArrowColumnVector ColumnVector ColumnarBatch TimeTrackingOutputStream JavaStreamingContext WriteAheadLog

closeWriter

ClusteringSummary Assignment scheduler

closureSerializer

SparkEnv

cloudWatchCredentials

Builder

cls

ObjectType

clsTag

Encoder

cluster

clusterCenters

BisectingKMeansModel KMeansModel BisectingKMeansModel KMeansModel StreamingKMeansModel

clusterSizes

ClusteringSummary

clusterWeights

StreamingKMeansModel

clustering

ml mllib

coalesce

JavaDoubleRDD JavaPairRDD JavaRDD PartitionCoalescer RDD Dataset functions

coefficientMatrix

LogisticRegressionModel

coefficientStandardErrors

GeneralizedLinearRegressionTrainingSummary LinearRegressionSummary

coefficients

LinearSVCModel LogisticRegressionModel AFTSurvivalRegressionModel GeneralizedLinearRegressionModel LinearRegressionModel

cogroup

JavaPairRDD PairRDDFunctions KeyValueGroupedDataset JavaPairDStream PairDStreamFunctions

col

Dataset functions

colIter

DenseMatrix Matrix SparseMatrix DenseMatrix Matrix SparseMatrix

colPtrs

SparseMatrix SparseMatrix

colRegex

colStats

Statistics

collect

JavaRDDLike EdgeRDDImpl RDD Dataset

collectAsList

collectAsMap

JavaRDDLike AsyncRDDActions

collectAsync

collectEdges

collectNeighborIds

collectNeighbors

collectPartitions

collectSubModels

collect_list

collect_set

collectionAccumulator

SparkContext

colsPerBlock

BlockMatrix

column

functions ColumnarBatch

columnSchema

ImageSchema

columnSimilarities

IndexedRowMatrix RowMatrix

columns

JavaPairRDD PairRDDFunctions JavaPairDStream PairDStreamFunctions

combineByKey

combineByKeyWithClassTag

PairRDDFunctions

combineCombinersByKey

Aggregator

combineValuesByKey

Aggregator

commit

ContinuousReader MicroBatchReader DataSourceWriter DataWriter StreamWriter

commitJob

FileCommitProtocol HadoopMapReduceCommitProtocol

commitTask

FileCommitProtocol HadoopMapReduceCommitProtocol SparkHadoopMapRedUtil

compare

Decimal RDDInfo

compileValue

JdbcDialect

completed

ApplicationAttemptInfo

completedTasks

ExecutorSummary

completionTime

StageInfo JobData StageData

compressed

Matrix Vector Vector

compressedColMajor

Matrix

compressedInputStream

CompressionCodec LZ4CompressionCodec LZFCompressionCodec SnappyCompressionCodec ZStdCompressionCodec

compressedOutputStream

CompressionCodec LZ4CompressionCodec LZFCompressionCodec SnappyCompressionCodec ZStdCompressionCodec

compressedRowMajor

Matrix

compute

computeColumnSummaryStatistics

computeCost

BisectingKMeansModel ~~KMeansModel~~ BisectingKMeansModel KMeansModel

computeCovariance

IndexedRowMatrix RowMatrix

computeError

Loss

computeGramianMatrix

computeInitialPredictionAndError

GradientBoostedTreesModel

computePreferredLocations

InputFormatInfo

computePrincipalComponents

computePrincipalComponentsAndExplainedVariance

IndexedRowMatrix RowMatrix

computeSVD

concat

concat_ws

SparkEnv SparkSession RelationConversions

conf

confidence

Rule BoundedDouble CountMinSketch

config

configuration

confusionMatrix

connectedComponents

consequent

contains

SparkConf ParamMap Column RuntimeConfig Metadata

containsDelimiters

HiveOptions

containsNull

ArrayType

context

ContextAwareIterator InterruptibleIterator JavaRDDLike ~~GeneralMLWriter~~ ~~MLReader~~ ~~MLWriter~~ RDD JavaDStreamLike DStream

conv

convertMatrixColumnsFromML

convertMatrixColumnsToML

convertToCanonicalEdges

GraphOps

convertVectorColumnsFromML

convertVectorColumnsToML

Estimator Model Pipeline PipelineModel PipelineStage Predictor Transformer UnaryTransformer DecisionTreeClassificationModel DecisionTreeClassifier GBTClassificationModel GBTClassifier LinearSVC LinearSVCModel LogisticRegression LogisticRegressionModel MultilayerPerceptronClassificationModel MultilayerPerceptronClassifier NaiveBayes NaiveBayesModel OneVsRest OneVsRestModel RandomForestClassificationModel RandomForestClassifier BisectingKMeans BisectingKMeansModel DistributedLDAModel GaussianMixture GaussianMixtureModel KMeans KMeansModel LDA LocalLDAModel PowerIterationClustering BinaryClassificationEvaluator ClusteringEvaluator Evaluator MulticlassClassificationEvaluator RegressionEvaluator Binarizer BucketedRandomProjectionLSH BucketedRandomProjectionLSHModel Bucketizer ChiSqSelector ChiSqSelectorModel CountVectorizer CountVectorizerModel FeatureHasher HashingTF IDF IDFModel Imputer ImputerModel IndexToString Interaction MaxAbsScaler MaxAbsScalerModel MinHashLSH MinHashLSHModel MinMaxScaler MinMaxScalerModel OneHotEncoder OneHotEncoderEstimator OneHotEncoderModel PCA PCAModel PolynomialExpansion QuantileDiscretizer RFormula RFormulaModel RegexTokenizer SQLTransformer StandardScaler StandardScalerModel StopWordsRemover StringIndexer StringIndexerModel Tokenizer VectorAssembler VectorIndexer VectorIndexerModel VectorSizeHint VectorSlicer Word2Vec Word2VecModel FPGrowth FPGrowthModel PrefixSpan DenseMatrix DenseVector Matrix SparseMatrix SparseVector Vector ParamMap Params ALS ALSModel AFTSurvivalRegression AFTSurvivalRegressionModel DecisionTreeRegressionModel DecisionTreeRegressor GBTRegressionModel GBTRegressor GeneralizedLinearRegression GeneralizedLinearRegressionModel IsotonicRegression IsotonicRegressionModel LinearRegression LinearRegressionModel RandomForestRegressionModel RandomForestRegressor CrossValidator CrossValidatorModel TrainValidationSplit TrainValidationSplitModel DenseMatrix DenseVector Matrix SparseMatrix SparseVector Vector ExponentialGenerator GammaGenerator LogNormalGenerator PoissonGenerator RandomDataGenerator StandardNormalGenerator UniformGenerator WeibullGenerator Strategy Row ColumnarArray ColumnarMap ColumnarRow AccumulatorV2 CollectionAccumulator DoubleAccumulator LegacyAccumulatorWrapper LongAccumulator StatCounter

copy

copyAndReset

AccumulatorV2 CollectionAccumulator

copyValues

Params

coresGranted

ApplicationInfo

coresPerExecutor

ApplicationInfo

corr

Correlation Statistics DataFrameStatFunctions functions

cos

cosh

JavaRDDLike EdgeRDDImpl VertexRDDImpl Summarizer MultivariateOnlineSummarizer MultivariateStatisticalSummary RDD Dataset KeyValueGroupedDataset RelationalGroupedDataset typed functions JavaDStreamLike DStream DoubleAccumulator LongAccumulator StatCounter

count

countApprox

JavaRDDLike RDD

countApproxDistinct

JavaRDDLike RDD

countApproxDistinctByKey

JavaRDDLike AsyncRDDActions

countAsync

countByKey

countByKeyApprox

JavaRDDLike RDD JavaDStreamLike DStream

countByValue

countByValueAndWindow

JavaDStreamLike DStream

countByValueApprox

JavaRDDLike RDD

countByWindow

JavaDStreamLike DStream

countDistinct

ExecutorLostFailure FetchFailed TaskCommitDenied TaskFailedReason TaskKilled

countMinSketch

DataFrameStatFunctions

countTowardsTaskFailures

cov

MultivariateGaussian DataFrameStatFunctions

covar_pop

covar_samp

crc32

create

JdbcRDD PartitionPruningRDD ~~ProcessingTime~~ RateEstimator

createCombiner

Aggregator

createCommitter

ContinuousReadSupport ContinuousInputPartition

createContinuousReader

createDataFrame

SQLContext SparkSession

createDataWriter

DataWriterFactory

createDataset

SQLContext SparkSession

~~createExternalTable~~

~~SQLContext~~ ~~Catalog~~

createGlobalTempView

createJobContext

LogisticRegressionWithLBFGS LogisticRegressionWithSGD SVMWithSGD GeneralizedLinearAlgorithm LassoWithSGD LinearRegressionWithSGD RidgeRegressionWithSGD

createMicroBatchReader

MicroBatchReadSupport

createModel

createOrReplaceGlobalTempView

createOrReplaceTempView

BucketedRandomProjectionLSH MinHashLSH

createPartitionReader

InputPartition

createRDDWithLocalProperties

DStream

createRawLSHModel

createReader

ReadSupport

createRelation

CreatableRelationProvider RelationProvider SchemaRelationProvider

createSink

StreamSinkProvider

createSource

StreamSourceProvider

~~createStream~~

~~KinesisUtils~~

createStreamWriter

StreamWriteSupport

createTable

Catalog

createTaskAttemptContext

createTempView

UnaryTransformer DCT ElementwiseProduct NGram Normalizer PolynomialExpansion RegexTokenizer Tokenizer

createTransformFunc

createUnsafe

createWriter

createWriterFactory

crossJoin

crosstab

DataFrameStatFunctions

csv

DataFrameReader DataFrameWriter DataStreamReader

cube

cume_dist

currentAttemptId

currentDatabase

currentRow

Window ~~functions~~

current_date

current_timestamp