org.apache.spark.mllib.clustering
Return the topics described by weighted terms.
Return the topics described by weighted terms.
Maximum number of terms to collect for each topic.
Array over topics. Each topic is represented as a pair of matching arrays: (term indices, term weights in topic). Each topic's terms are sorted in order of decreasing weight.
Return the topics described by weighted terms.
Return the topics described by weighted terms.
WARNING: If vocabSize and k are large, this can return a large object!
Array over topics. Each topic is represented as a pair of matching arrays: (term indices, term weights in topic). Each topic's terms are sorted in order of decreasing weight.
Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").
Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").
This is the parameter to a Dirichlet distribution.
Current version of model save/load format.
Current version of model save/load format.
Shape parameter for random initialization of variational parameter gamma.
Shape parameter for random initialization of variational parameter gamma. Used for variational inference for perplexity and other test-time computations.
Number of topics
Number of topics
Java-friendly version of logLikelihood
Java-friendly version of logLikelihood
Calculates a lower bound on the log likelihood of the entire corpus.
Calculates a lower bound on the log likelihood of the entire corpus.
See Equation (16) in original Online LDA paper.
test corpus to use for calculating log likelihood
variational lower bound on the log likelihood of the entire corpus
Java-friendly version of logPerplexity
Java-friendly version of logPerplexity
Calculate an upper bound bound on perplexity.
Calculate an upper bound bound on perplexity. (Lower is better.) See Equation (16) in original Online LDA paper.
test corpus to use for calculating perplexity
Variational upper bound on log perplexity per token.
Save this model to the given path.
Save this model to the given path.
This saves:
The model may be loaded using Loader.load.
Spark context used to save model data.
Path specifying the directory in which to save this model. If the directory already exists, this method throws an exception.
Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics' distributions over terms.
Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics' distributions over terms.
This is the parameter to a symmetric Dirichlet distribution.
Note: The topics' distributions over terms are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many later papers such as Asuncion et al., 2009.
Java-friendly version of topicDistributions
Java-friendly version of topicDistributions
Predicts the topic mixture distribution for each document (often called "theta" in the literature).
Predicts the topic mixture distribution for each document (often called "theta" in the literature). Returns a vector of zeros for an empty document.
This uses a variational approximation following Hoffman et al. (2010), where the approximate distribution is called "gamma." Technically, this method returns this approximation "gamma" for each document.
documents to predict topic mixture distributions for
An RDD of (document ID, topic mixture distribution for document)
Inferred topics (vocabSize x k matrix).
Inferred topics (vocabSize x k matrix).
Inferred topics, where each topic is represented by a distribution over terms.
Inferred topics, where each topic is represented by a distribution over terms. This is a matrix of size vocabSize x k, where each column is a topic. No guarantees are given about the ordering of the topics.
Vocabulary size (number of terms or terms in the vocabulary)
Vocabulary size (number of terms or terms in the vocabulary)
Local LDA model. This model stores only the inferred topics.