In a Cloudera Spark-on-YARN installation, I encounter an application that has multiple log4j APIs in its classpath:
- log4j:1.2
- log4j-1.2-api:2.x
- log4j-core:2.x
When the application is launched. I found the following error:
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
It cannot be fixed because the vendor of the platform exclusively used the log4j:1.2 configuration everywhere: it has integrated a managed log4j.properties file (not the log4j2.properties), which is in the classpath, used by many components, and cannot be tampered.
How do I tell the classloader or log4j implementation to not trying to find log4j2 file if it is missing?
UPDATE 1: I've tried the following Spark arguments to customise the driver JVM. Effects are as follows:
--conf spark.driver.extraJavaOptions="-XX:+UseG1GC -Dlog4j1.compatibility=true -Dlog4j.configuration=./conf_5/log4j.properties -Dlog4j.debug"
result is:
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
log4j: setFile called: /tmp/spark-b190d9df-64bf-4ff6-b72d-2cb8de1d888a/__driver_logs__/driver.log, true
log4j: setFile ended
log4j: Finalizing appender named [_DriverLogAppender].
--conf spark.driver.extraJavaOptions="-XX:+UseG1GC -Dlog4j1.compatibility=true -Dlog4j.configurationFile=./conf_5/log4j.properties -Dlog4j.debug"
result is:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
log4j: setFile called: /tmp/spark-817bc6eb-bcc0-4cee-ba32-dbd4d89048e5/__driver_logs__/driver.log, true
log4j: setFile ended
log4j: Finalizing appender named [_DriverLogAppender].
So none of them is working properly. There seems to be no way to either use the right version or use the wrong configuration file