What means "package url -> 3.1.1" in the runtime classpath?

user_2021516 0
Viewed 21

What means "org.apache.hadoop:hadoop-mapreduce-client-core:2.6.5 -> 3.1.1" in the classpath below? Spark 2.4.7 will use version 3.1.1 of hadoop-mapreduce-client-core?

$ gradlew dependencies --configuration runtimeClasspath

> Task :dependencies

runtimeClasspath - Runtime classpath of source set 'main'.
\--- org.apache.spark:spark-core_2.11:2.4.7
     +--- org.apache.hadoop:hadoop-client:2.6.5
     |    +--- org.apache.hadoop:hadoop-common:2.6.5
     |    |    +--- org.apache.hadoop:hadoop-annotations:2.6.5 -> 3.1.1
     |    |    +--- com.google.guava:guava:11.0.2 -> 16.0.1
     |    +--- org.apache.hadoop:hadoop-mapreduce-client-app:2.6.5
     |    |    +--- org.apache.hadoop:hadoop-mapreduce-client-common:2.6.5
     |    |    |    +--- org.apache.hadoop:hadoop-yarn-common:2.6.5 -> 3.1.1 (*)
     |    +--- org.apache.hadoop:hadoop-mapreduce-client-core:2.6.5 -> 3.1.1 (*)

(*) - dependencies omitted (listed previously)
apache-spark gradle java
1 Answers

The answer is: Spark 2.4.7 will use version 3.1.1 of hadoop-mapreduce-client-core

"Gradle resolves any dependency version conflicts by selecting the latest version found in the dependency graph"

https://docs.gradle.org/current/userguide/dependency_downgrade_and_exclude.html

E.G:

https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11/2.4.7

depends on:

  • org.apache.commons:commons-math3:3.4.1
  • org.apache.hadoop:hadoop-common:2.6.5

org.apache.hadoop:hadoop-common:2.6.5 depends on:

  • org.apache.commons:commons-math3:3.1.1

The result, gradle will use org.apache.commons:commons-math3:3.4.1 for both package.

user_20215160
Related