Spark and Elastic causing jackson overlap

user_4648873 0
Viewed 111

We recently upgraded spark to 2.2.0 in our application and this brings in Jackson 2.6.5 jars in the runtime.

But we have other services that need Jackson 2.8.6(due to Elastic 5.5.0) the service has a manifest which refers to Jackson 2.8.6 from the application's lib folder.

So there are two versions of Jackson jars loaded in the runtime and spark is picking a wrong one (2.8.6 from lib folder instead of 2.6.5 from its own jars folder)

On windows the version picked is the right one although both versions are still loaded in the runtime.

But it fails on Linux.

How do we resolve this?

elasticsearch apache-spark jackson classpath java
1 Answers

You can add jars to the classpath programatically using addjar, inside file spark-defaults.conf or with spark-submit flags.

Also, We have 2 properties from runtime-environment docs here to mitigate conflicts

spark.driver.userClassPathFirst

default false (Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in the driver. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. Note : It is currently an experimental feature. This is used in cluster mode only.

spark.executor.userClassPathFirst

default false (Experimental) Same functionality as spark.driver.userClassPathFirst, but applied to executor instances.

Properties set directly on the SparkConf take highest precedence, then flags passed to spark-submit or spark-shell, then options in the spark-defaults.conf file

user_6470530
Related