Migration Guide: PySpark (Python on Spark)

• Upgrading from PySpark 2.4 to 3.0
• Upgrading from PySpark 2.3 to 2.4
• Upgrading from PySpark 2.3.0 to 2.3.1 and above
• Upgrading from PySpark 2.2 to 2.3
• Upgrading from PySpark 1.4 to 1.5
• Upgrading from PySpark 1.0-1.2 to 1.3

Note that this migration guide describes the items specific to PySpark. Many items of SQL migration can be applied when migrating PySpark to higher versions. Please refer Migration Guide: SQL, Datasets and DataFrame.

Upgrading from PySpark 2.4 to 3.0

• Since Spark 3.0, PySpark requires a Pandas version of 0.23.2 or higher to use Pandas related functionality, such as toPandas, createDataFrame from Pandas DataFrame, etc.

• Since Spark 3.0, PySpark requires a PyArrow version of 0.12.1 or higher to use PyArrow related functionality, such as pandas_udf, toPandas and createDataFrame with “spark.sql.execution.arrow.enabled=true”, etc.

• In PySpark, when creating a SparkSession with SparkSession.builder.getOrCreate(), if there is an existing SparkContext, the builder was trying to update the SparkConf of the existing SparkContext with configurations specified to the builder, but the SparkContext is shared by all SparkSessions, so we should not update them. Since 3.0, the builder comes to not update the configurations. This is the same behavior as Java/Scala API in 2.3 and above. If you want to update them, you need to update them prior to creating a SparkSession.

• In PySpark, when Arrow optimization is enabled, if Arrow version is higher than 0.11.0, Arrow can perform safe type conversion when converting Pandas.Series to Arrow array during serialization. Arrow will raise errors when detecting unsafe type conversion like overflow. Setting spark.sql.execution.pandas.arrowSafeTypeConversion to true can enable it. The default setting is false. PySpark’s behavior for Arrow versions is illustrated in the table below:

  PyArrow version	Integer Overflow	Floating Point Truncation
  version < 0.11.0	Raise error	Silently allows
  version > 0.11.0, arrowSafeTypeConversion=false	Silent overflow	Silently allows
  version > 0.11.0, arrowSafeTypeConversion=true	Raise error	Raise error

• Since Spark 3.0, createDataFrame(..., verifySchema=True) validates LongType as well in PySpark. Previously, LongType was not verified and resulted in None in case the value overflows. To restore this behavior, verifySchema can be set to False to disable the validation.

• Since Spark 3.0, Column.getItem is fixed such that it does not call Column.apply. Consequently, if Column is used as an argument to getItem, the indexing operator should be used. For example, map_col.getItem(col('id')) should be replaced with map_col[col('id')].

Upgrading from PySpark 2.3 to 2.4

• In PySpark, when Arrow optimization is enabled, previously toPandas just failed when Arrow optimization is unable to be used whereas createDataFrame from Pandas DataFrame allowed the fallback to non-optimization. Now, both toPandas and createDataFrame from Pandas DataFrame allow the fallback by default, which can be switched off by spark.sql.execution.arrow.fallback.enabled.

Upgrading from PySpark 2.3.0 to 2.3.1 and above

• As of version 2.3.1 Arrow functionality, including pandas_udf and toPandas()/createDataFrame() with spark.sql.execution.arrow.enabled set to True, has been marked as experimental. These are still evolving and not currently recommended for use in production.

Upgrading from PySpark 2.2 to 2.3

• In PySpark, now we need Pandas 0.19.2 or upper if you want to use Pandas related functionalities, such as toPandas, createDataFrame from Pandas DataFrame, etc.

• In PySpark, the behavior of timestamp values for Pandas related functionalities was changed to respect session timezone. If you want to use the old behavior, you need to set a configuration spark.sql.execution.pandas.respectSessionTimeZone to False. See SPARK-22395 for details.

• In PySpark, na.fill() or fillna also accepts boolean and replaces nulls with booleans. In prior Spark versions, PySpark just ignores it and returns the original Dataset/DataFrame.

• In PySpark, df.replace does not allow to omit value when to_replace is not a dictionary. Previously, value could be omitted in the other cases and had None by default, which is counterintuitive and error-prone.

Upgrading from PySpark 1.4 to 1.5

• Resolution of strings to columns in Python now supports using dots (.) to qualify the column or access nested values. For example df['table.column.nestedField']. However, this means that if your column name contains any dots you must now escape them using backticks (e.g., table.`column.with.dots`.nested).

• DataFrame.withColumn method in PySpark supports adding a new column or replacing existing columns of the same name.

Upgrading from PySpark 1.0-1.2 to 1.3

Python DataTypes No Longer Singletons

When using DataTypes in Python you will need to construct them (i.e. StringType()) instead of referencing a singleton.