org.apache.spark#hadoop-cloud_2.12;3.1.2: not found

user_1088380 0
Viewed 642

On a Spark SBT project, I follow https://spark.apache.org/docs/latest/cloud-integration.html to access data directly on s3. However, I get org.apache.spark#hadoop-cloud_2.12;3.1.2: not found. The docs mention provided scope but as this job is being run locally, there's no JDK/application container providing anything. On Maven Central I see various vendors publishing with their respective version schemes, but none matching my specific Spark version: 3.1.2

These are my relevant dependencies:

  // Version.spark is 3.1.2
  val spark   = Seq(
    "org.apache.spark" %% "spark-core"   % Version.spark,
    "org.apache.spark" %% "spark-sql"    % Version.spark,
    "org.apache.spark" %% "hadoop-cloud" % Version.spark
  )

Should I instead fall back onto hadoop-aws packages? https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html

apache-spark hadoop scala
2 Answers

Like the hadoop-aws dependency, it is not included and at this point the latest jar is located at cloudera repository which is not added by default to spark. https://mvnrepository.com/artifact/org.apache.spark/spark-hadoop-cloud?repo=cloudera-repos

The following worked for me with spark 3.1.2:

.config("spark.jars.repositories", "https://repository.cloudera.com/artifactory/cloudera-repos/")
.config("spark.jars.packages", "org.apache.spark:spark-hadoop-cloud_2.12:3.1.1.3.1.7270.0-253")
user_39225170

In the build.sbt file you can add the following lines

libraryDependencies ++= Seq("org.apache.spark" %% "spark-hadoop-cloud" % "3.1.1.3.1.7270.0-253")


resolvers ++= Seq("Cloudera spark-cloud" at "https://repository.cloudera.com/artifactory/cloudera-repos/")

Please note that the %% in libraryDependencies adds your project's scala version to the artifact name. Read more here.

user_37224790
Related