Migration Guides

This page includes links to guides that will help you migrate to PySpark.

If you are upgrading from an older to a newer version of PySpark, refer to the following page for differences between versions:

• Upgrading PySpark

If you are migrating code from Koalas to PySpark, this guide is for you:

• Migrating from Koalas to pandas API on Spark

A lot of content in other migration guides can also be helpful when migrating to newer PySpark versions because PySpark internally shares the same components. Please also refer to the following migration guides:

• Migration Guide: SQL, Datasets and DataFrame

• Migration Guide: Structured Streaming

• Migration Guide: MLlib (Machine Learning)

• Migration Guide: Spark Core