pyspark.sql.functions.flatten

pyspark.sql.functions.flatten(col)

Collection function: creates a single array from an array of arrays. If a structure of nested arrays is deeper than two levels, only one level of nesting is removed.

New in version 2.4.0.

Parameters

colColumn or str

name of column or expression

Examples

>>> df = spark.createDataFrame([([[1, 2, 3], [4, 5], [6]],), ([None, [4, 5]],)], ['data'])
>>> df.select(flatten(df.data).alias('r')).collect()
[Row(r=[1, 2, 3, 4, 5, 6]), Row(r=None)]

pyspark.sql.functions.first pyspark.sql.functions.floor