Multiple driver-java-options in spark submit

user_185572 0
Viewed 18802

I am using a spark-submit specified in a bash script as:

CLUSTER_OPTIONS=" \
--master yarn-cluster \
--files     file:///${CONF_DIR}/app.conf#app.conf,file:///${CONF_DIR}/log4j-executor.xml#log4j.xml \
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.xml" \
--driver-java-options '-Dlog4j.configuration=file:log4j.xml -Dconfig.file=app.conf' \
--keytab ${KEYTAB} \
--principal ${PRINCIPAL} \
"

I am finding that app conf is not being picked up as I receive this error:

Error: Unrecognized option: -Dconfig.file=file:app.conf'

I have also attempted different ways to encapsulate the driver-java-options:

1) 

--driver-java-options \"-Dlog4j.configuration=file:log4j.xml -Dconfig.file=app.conf\" \

Error: Unrecognized option: -Dconfig.file=file:app.conf"

2) 

--driver-java-options "-Dlog4j.configuration=file:log4j.xml -Dconfig.file=file:transformation.conf" \


./start_app.sh: line 30: -Dconfig.file=file:app.conf --keytab /app/conf/keytab/principal.keytab --principal principal : No such file or directory

How can i specify multiple driver-java-options for use by my Spark app?

N.B. I am using Spark 1.5.0

apache-spark bash
4 Answers

Just writing this because it was so odd. The way I got this to work, it was not until I made --driver-java-options the first of all arguments. I left it as is so you get the entirety.

Using pyspark Local mode

/opt/apache-spark/spark-2.3.0-bin-hadoop2.7/bin/spark-submit \
    --driver-java-options "-Xms2G -Doracle.jdbc.Trace=true -Djava.util.logging.config.file=/opt/apache-spark/spark-2.3.0-bin-hadoop2.7/conf/oraclejdbclog.properties -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=1098 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=192.168.2.120 -Dcom.sun.management.jmxremote.rmi.port=1095" \
    --driver-memory $_driver_memory \
    --executor-memory $_executor_memory \
    --total-executor-cores $_total_executor_cores \
    --verbose \
    --jars /opt/apache-spark/jars/log4j-1.2.17.jar main.py \
    --dbprefix $1 \
    --copyfrom $2

Hope this helps someone.

user_30084100

Working 2018 in Spark 2.3.0

spark2-submit \
--class com.demo.Main \
--master yarn --deploy-mode client \
--driver-memory 10G --driver-cores 8 --executor-memory 13G --executor-cores 4 \
--num-executors 10 \
--verbose \
--conf "spark.driver.extraJavaOptions=-Dconfig.file=/$HOME/application.conf -Dlog4j.configuration=$HOME/log4j.properties" \
--conf "spark.executor.extraJavaOptions=-Dconfig.file=$HOME/application.conf -Dlog4j.configuration=$HOME/log4j.properties" \
--files "application.conf,log4j.properties" \
$HOME/ingestion-1.0-SNAPSHOT-jar-with-dependencies.jar
user_22833140

On Spark 2.4.5, further complicated by Docker and env var passing through to an in-container spark-submit wrapper:

SPARK_SUBMIT_ARGS="--conf spark.driver.host=host.docker.internal --conf spark.driver.port=46111"

The spark-submit wrapper container then executed like so:

docker run \
    -e SPARK_SUBMIT_ARGS="${SPARK_SUBMIT_ARGS}" \
    $IMAGE_NAME

where SPARK_SUBMIT_ARGS env var is referenced by the spark-submit wrapper script.

Sample run output, with configuration being dumped out by the driver code:

20/09/15 08:02:13 INFO SparkContext: Running Spark version 2.4.5
20/09/15 08:02:13 INFO SparkContext: Submitted application: rule-query-app
20/09/15 08:02:13 INFO SparkContext: Spark configuration:
spark.app.name=rule-query-app
spark.driver.host=host.docker.internal
spark.driver.port=46111
spark.jars=file:/app/rule-query-spark-app-fat.jar
spark.logConf=true
spark.master=spark://host.docker.internal:7077
spark.submit.deployMode=client

Kudos to this answer (upvoted, too!) providing the --conf hint.

user_39960580
Related