pyspark.inheritable_thread_target

pyspark.inheritable_thread_target(f)

Return thread target wrapper which is recommended to be used in PySpark when the pinned thread mode is enabled. The wrapper function, before calling original thread target, it inherits the inheritable properties specific to JVM thread such as InheritableThreadLocal.

Also, note that pinned thread mode does not close the connection from Python to JVM when the thread is finished in the Python side. With this wrapper, Python garbage-collects the Python thread instance and also closes the connection which finishes JVM thread correctly.

When the pinned thread mode is off, it return the original f.

New in version 3.2.0.

Parameters

ffunction

the original thread target.

Notes

This API is experimental.

It is important to know that it captures the local properties when you decorate it whereas InheritableThread captures when the thread is started. Therefore, it is encouraged to decorate it when you want to capture the local properties.

For example, the local properties from the current Spark context is captured when you define a function here instead of the invocation:

>>> @inheritable_thread_target
... def target_func():
...     pass  # your codes.

If you have any updates on local properties afterwards, it would not be reflected to the Spark context in target_func().

The example below mimics the behavior of JVM threads as close as possible:

>>> Thread(target=inheritable_thread_target(target_func)).start()  

pyspark.AccumulatorParam.zero pyspark.SparkConf.contains