Setting Environment variables in Spark Cluster Mode

user_9019448 0
Viewed 6689

I was going through this Apache Spark documentation, and it mentions that:

When running Spark on YARN in cluster mode, environment variables need to be set using the spark.yarn.appMasterEnv.[EnvironmentVariableName] property in your conf/spark-defaults.conf file.

I am running my EMR cluster on AWS data pipeline. I wanted to know that where do I have to edit this conf file. Also, if I create my own custom conf file, and specify it as part of --configurations (in the spark-submit), will it solve my use-case?

hadoop-yarn apache-spark environment-variables
2 Answers

For future reference you could directly pass the environment variable when creating the EMR cluster using the Configurations parameter as described in the docs here.

Specifically, the spark-defaults file can be modified by passing a configuration JSON as follows:

{
    'Classification': 'spark-defaults',
    'Properties': {
        'spark.yarn.appMasterEnv.[EnvironmentVariableName]' = 'some_value',
        'spark.executorEnv.[EnvironmentVariableName]': 'some_other_value'
    }
},

Where spark.yarn.appMasterEnv.[EnvironmentVariableName] would be used to pass a variable in cluster mode using YARN (here). And spark.executorEnv.[EnvironmentVariableName] to pass a variable to the executor process (here).

user_118099110
Related