pyspark.RDD.randomSplit

RDD.randomSplit(weights: Sequence[Union[int, float]], seed: Optional[int] = None) → List[pyspark.rdd.RDD[T]]

Randomly splits this RDD with the provided weights.

weightslist

weights for splits, will be normalized if they don’t sum to 1

seedint, optional

random seed

Returns

list

split RDDs in a list

Examples

>>> rdd = sc.parallelize(range(500), 1)
>>> rdd1, rdd2 = rdd.randomSplit([2, 3], 17)
>>> len(rdd1.collect() + rdd2.collect())
500
>>> 150 < rdd1.count() < 250
True
>>> 250 < rdd2.count() < 350
True

pyspark.RDD.pipe pyspark.RDD.reduce