pyspark.SparkContext.setJobDescription

SparkContext.setJobDescription(value)

Set a human readable description of the current job.

Notes

Currently, setting a job description (set to local properties) with multiple threads does not properly work. Internally threads on PVM and JVM are not synced, and JVM thread can be reused for multiple threads on PVM, which fails to isolate local properties for each thread on PVM.

To avoid this, enable the pinned thread mode by setting PYSPARK_PIN_THREAD environment variable to true and uses pyspark.InheritableThread.

pyspark.SparkContext.setCheckpointDir pyspark.SparkContext.setJobGroup