pyspark.sql.DataFrame.pandas_api¶
-
DataFrame.pandas_api(index_col: Union[str, List[str], None] = None) → PandasOnSparkDataFrame[source]¶ Converts the existing DataFrame into a pandas-on-Spark DataFrame.
New in version 3.2.0.
Changed in version 3.5.0: Supports Spark Connect.
If a pandas-on-Spark DataFrame is converted to a Spark DataFrame and then back to pandas-on-Spark, it will lose the index information and the original index will be turned into a normal column.
This is only available if Pandas is installed and available.
- Parameters
- index_col: str or list of str, optional, default: None
Index column of table in Spark.
- Returns
PandasOnSparkDataFrame
See also
pyspark.pandas.frame.DataFrame.to_spark
Examples
>>> df = spark.createDataFrame( ... [(14, "Tom"), (23, "Alice"), (16, "Bob")], ["age", "name"])
>>> df.pandas_api() age name 0 14 Tom 1 23 Alice 2 16 Bob
We can specify the index columns.
>>> df.pandas_api(index_col="age") name age 14 Tom 23 Alice 16 Bob