org.apache.spark.mllib.stat.test

Class ChiSqTest

Object

org.apache.spark.mllib.stat.test.ChiSqTest

All Implemented Interfaces:

Logging

public class ChiSqTest
extends Object
implements Logging

Conduct the chi-squared test for the input RDDs using the specified method. Goodness-of-fit test is conducted on two Vectors, whereas test of independence is conducted on an input of type Matrix in which independence between columns is assessed. We also provide a method for computing the chi-squared statistic between each feature and the label for an input RDD[LabeledPoint], return an Array[ChiSquaredTestResult] of size = number of features in the input RDD.

Supported methods for goodness of fit: pearson (default) Supported methods for independence: pearson (default)

More information on Chi-squared test: http://en.wikipedia.org/wiki/Chi-squared_test

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ChiSqTest.Method
static class	ChiSqTest.Method$
static class	ChiSqTest.NullHypothesis$

Constructor Summary

Constructors

Constructor and Description
ChiSqTest()

Method Summary

Methods

Modifier and Type	Method and Description
static ChiSqTestResult	chiSquared(Vector observed, Vector expected, String methodName)
static ChiSqTestResult[]	chiSquaredFeatures(RDD<LabeledPoint> data, String methodName) Conduct Pearson's independence test for each feature against the label across the input RDD.
static ChiSqTestResult	chiSquaredMatrix(Matrix counts, String methodName)
static ChiSqTest.Method	PEARSON()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

ChiSqTest

public ChiSqTest()

Method Detail

PEARSON

public static ChiSqTest.Method PEARSON()

chiSquaredFeatures

public static ChiSqTestResult[] chiSquaredFeatures(RDD<LabeledPoint> data,
                                   String methodName)

Conduct Pearson's independence test for each feature against the label across the input RDD. The contingency table is constructed from the raw (feature, label) pairs and used to conduct the independence test. Returns an array containing the ChiSquaredTestResult for every feature against the label.

chiSquared

public static ChiSqTestResult chiSquared(Vector observed,
                         Vector expected,
                         String methodName)

chiSquaredMatrix

public static ChiSqTestResult chiSquaredMatrix(Matrix counts,
                               String methodName)