pyspark.sql.DataFrame.toPandas

DataFrame.toPandas() → PandasDataFrameLike

Returns the contents of this DataFrame as Pandas pandas.DataFrame.

This is only available if Pandas is installed and available.

New in version 1.3.0.

Changed in version 3.4.0: Supports Spark Connect.

Notes

This method should only be used if the resulting Pandas pandas.DataFrame is expected to be small, as all the data is loaded into the driver’s memory.

Usage with spark.sql.execution.arrow.pyspark.enabled=True is experimental.

Examples

>>> df.toPandas()  
   age   name
0    2  Alice
1    5    Bob

pyspark.sql.DataFrame.toLocalIterator pyspark.sql.DataFrame.to_pandas_on_spark