Dataproc: how do I configure Spark driver and executor log4j properties?

user_1716566 0
Viewed 1844

As explained in previous answers, the ideal way to change the verbosity of a Spark cluster is changing the corresponding log4j.properties. However, on dataproc Spark runs on Yarn, therefore we have to adjust the global configuration and not /usr/lib/spark/conf

Several suggestions:

On dataproc we have several gcloud commands and properties we can pass during cluster creation. See documentation Is it possible to change the log4j.properties under /etc/hadoop/conf by specifying

--properties 'log4j:hadoop.root.logger=WARN,console'

Maybe not, as from the docs:

The --properties command cannot modify configuration files not shown above.

Another way would be to use a shell script during cluster init and run sed:

# change log level for each node to WARN
sudo sed -i -- 's/log4j.rootCategory=INFO, console/log4j.rootCategory=WARN, console/g'\
                     /etc/spark/conf/log4j.properties
sudo sed -i -- 's/hadoop.root.logger=INFO,console/hadoop.root.logger=WARN,console/g'\
                    /etc/hadoop/conf/log4j.properties

But is it enough or do we need to change the env variable hadoop.root.logger as well?

google-cloud-dataproc logging
3 Answers
user_12096860

Updated in Q3 2022

Default config

The default log4j config for Spark on Dataproc is available at /etc/spark/conf/log4j.properties. It configures root logger to stderr at INFO level. But at runtime driver logs (in client mode) will be directed by the Dataproc agent to GCS and streamed back to the client, and executor logs (and driver logs in cluster mode) will be redirected by YARN to the stderr file in the container's YARN log dir. See this answer for how to get YARN container logs of Dataproc.

Consider using /etc/spark/conf/log4j.properties as the template for your custom config, and keep using console as the target for your log.

Cluster level

If you want to configure Spark driver and executor logs at cluster level, the simplest way is to add --properties spark-log4j:<key>=<value>,... when creating the cluster. The properties from the flag will be appended to /etc/spark/conf/log4j.properties which will be used as the default log4j config for all Spark drivers and executors in the cluster. Or you can write an init action to update the file.

Job level

You can also configure Spark driver and/or executor logs at job level when submitting the job with either of the following ways:

  1. --driver-log-levels (for driver only), for example:
gcloud dataproc jobs submit spark ...\
  --driver-log-levels root=WARN,org.apache.spark=DEBUG
  1. --files and --properties spark.[driver|executor].extraJavaOptions=-Dlog4j.configuration= (for both driver and executor), for example:
gcloud dataproc jobs submit spark ... \
  --files gs://my-bucket/driver-log4j.properties,gs://my-bucket/executor-log4j.properties \
  --properties 'spark.driver.extraJavaOptions=-Dlog4j.configuration=file:driver-log4j.properties,spark.executor.extraJavaOptions=-Dlog4j.configuration=file:executor-log4j.properties'

Note that -Dlog4j.configuration should be set to file:<filename> because the files will be present in the working directory of the YARN container for driver/executor.

See also https://spark.apache.org/docs/latest/running-on-yarn.html#debugging-your-application

user_4042640
Related