Warnings while building Scala/Spark project with SBT

Viewed 7317

I'm trying to build a Scala/Spark project in IntelliJ Idea with the following build.sbt:

name := "try"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "2.2.0"

resolvers ++= Seq(
  "apache-snapshots" at "http://repository.apache.org/snapshots/"
)

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-mllib" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-hive" % sparkVersion
)

and getting a bunch of warnings:

8/6/17
1:29 PM SBT project import
                [warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
                [warn]  * io.netty:netty:3.9.9.Final is selected over {3.6.2.Final, 3.7.0.Final}
                [warn]      +- org.apache.spark:spark-core_2.11:2.2.0             (depends on 3.9.9.Final)
                [warn]      +- org.apache.zookeeper:zookeeper:3.4.6               (depends on 3.6.2.Final)
                [warn]      +- org.apache.hadoop:hadoop-hdfs:2.6.5                (depends on 3.6.2.Final)
                [warn]  * commons-net:commons-net:2.2 is selected over 3.1
                [warn]      +- org.apache.spark:spark-core_2.11:2.2.0             (depends on 2.2)
                [warn]      +- org.apache.hadoop:hadoop-common:2.6.5              (depends on 3.1)
                [warn]  * com.google.guava:guava:11.0.2 is selected over {12.0.1, 16.0.1}
                [warn]      +- org.apache.hadoop:hadoop-yarn-client:2.6.5         (depends on 11.0.2)
                [warn]      +- org.apache.hadoop:hadoop-yarn-api:2.6.5            (depends on 11.0.2)
                [warn]      +- org.apache.hadoop:hadoop-yarn-common:2.6.5 

I have several, perhaps dumb, questions:

  1. Is there a better way to structure build.sbt (add other resolvers e.g.?), so that I can get rid off the warnings?
  2. Should I care about warnings at all?
3 Answers

If everything works fine, you can disable these warnings by adding this to build settings:

evictionWarningOptions in update := EvictionWarningOptions.default
.withWarnTransitiveEvictions(false)
Related