Use case of spark.executor.allowSparkContext

user_8392616 0
Viewed 1391

I'm looking into spark-core, I found one undocumented config, which is spark.executor.allowSparkContext available since 3.0.1. I wasn't able to find detail in spark official documentation. In code, there is short description for this config

If set to true, SparkContext can be created in executors.

But I wonder that, How can SparkContext be created in executors? As far as I know SparkContext is created on driver, and executors are assigned by resource manager. So SparkContext is always created before executors. What is the use case of this config?

pyspark apache-spark
2 Answers

From the Spark Core migration 3.0 to 3.1:

In Spark 3.0 and below, SparkContext can be created in executors. Since Spark 3.1, an exception will be thrown when creating SparkContext in executors. You can allow it by setting the configuration spark.executor.allowSparkContext when creating SparkContext in executors.

As per this issue SPARK-32160, since version 3.1 there is a check added when creating SparkContext (see for pyspark pyspark/context.py) which prevents executors from creating SparkContext:

if (conf is None or
        conf.get("spark.executor.allowSparkContext", "false").lower() != "true"):
   # In order to prevent SparkContext from being created in executors.
   SparkContext._assert_on_driver() 

# ...

@staticmethod
def _assert_on_driver():
    """
    Called to ensure that SparkContext is created only on the Driver.
    Throws an exception if a SparkContext is about to be created in executors.
    """
    if TaskContext.get() is not None:
        raise Exception("SparkContext should only be created and accessed on the driver.")
user_13865510

An error in the docs and, or implementation I suggest.

The whole concept makes no sense if you (as you do) understand the Spark architecture. No announcement has been made otherwise about this.

From the other answer and plentiful doc of errors on this aspect it is clear something went awry.

user_69339930
Related