I had a nice little Maven-built CLI application using Java 17 on Windows 10. For logging I went over SLF4J (org.slf4j:slf4j-api:1.7.36) with Logback (ch.qos.logback:logback-classic:jar:1.2.11) as the implementation.
That was fine until I added Spark 3.3.0 into the mix:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
Then when I run my application, I get this:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in […/lib/logback-classic-1.2.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in […/lib/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
When I do a mvn dependency:tree, it looks like Spark is bringing in all this mess:
[INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.36:compile
[INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.36:compile
[INFO] | +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.17.2:compile
[INFO] | +- org.apache.logging.log4j:log4j-api:jar:2.17.2:compile
[INFO] | +- org.apache.logging.log4j:log4j-core:jar:2.17.2:compile
[INFO] | +- org.apache.logging.log4j:log4j-1.2-api:jar:2.17.2:compile
In particular org.apache.logging.log4j:log4j-slf4j-impl:jar:2.17.2 is the core problem; it's a Log4j 2 SLF4J Binding for an application logging to SLF4J, to use Log4J as the SLF4J implementation. But I'm using Logback as my SLF4J implementation. So I can exclude the Log4J implementation like this:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
</exclusion>
</exclusions>
</dependency>
Wonderful; now the warning goes away. I've solved the problem.
But here is my concern: you can see above that Spark is also bringing in log4j-api (the Log4J API itself) as well as log4j-1.2-api, which is a Log4j 1.2 Bridge for applications that write to the Log4J 1.x API! Does this mean that Spark is writing to the Log4J 2.x API itself, or even (horrors!) to the Log4J 1.x API? ♂️
Do I also need to include a Log4J-to-SLF4J bridge dependency so that whatever code that uses Log4J in some corner of Spark will correctly get its log output routed to SLF4J? Or is all this extra Log4J gunk included because someone setting up the POM didn't quite fully understand what it means to write to the SLF4J API, and just threw the kitchen sink at things?
In other words, am I good with the Log4J 2 implementation exclusion above, or do I also need to add Log4J-to-SLF4j bridges for some piece of Spark that writes directly to the Log4J API rather than through the SLF4J API?