intellij: errors with scala, spark

Mandroid 0
Viewed 26

I have a scala project using spark libraies, and it works fine most of the times(using intellij). But some times it starts giving errors on intellij launch:

    [warn]  Note: Unresolved dependencies path:
    [error] stack trace is suppressed; run 'last update' for the full output
    [error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
    [error] (update) sbt.librarymanagement.ResolveException: Error downloading org.apache.spark:spark-core_2.12:3.0.0-preview2
    [error]   Not found
    [error]   Not found
    [error]   not found: C:\...\.ivy2\localorg.apache.spark\spark-core_2.12\3.0.0-preview2\ivys\ivy.xml
    [error]   checksum format error: C:\....\AppData\Local\Coursier\Cache\v1\https\repo1.maven.org\maven2\org\apache\spark\spark-core_2.12\3.0.0-preview2\.spark-core_2.12-3.0.0-preview2.pom__sha1
    [error]   checksum format error: C:\....\AppData\Local\Coursier\Cache\v1\https\repo1.maven.org\maven2\org\apache\spark\spark-core_2.12\3.0.0-preview2\.spark-core_2.12-3.0.0-preview2.pom__sha1
    [error] Error downloading org.apache.spark:spark-sql_2.12:3.0.0-preview2
    [error]   Not found
    [error]   Not found
    [error]   not found: C:\...\.ivy2\localorg.apache.spark\spark-sql_2.12\3.0.0-preview2\ivys\ivy.xml
    [error]   checksum format error: C:\....\AppData\Local\Coursier\Cache\v1\https\repo1.maven.org\maven2\org\apache\spark\spark-sql_2.12\3.0.0-preview2\.spark-sql_2.12-3.0.0-preview2.pom__sha1
    [error] (ssExtractDependencies) sbt.librarymanagement.ResolveException: Error downloading org.apache.spark:spark-core_2.12:3.0.0-preview2
    [error]   Not found
    [error]   Not found
    [error]   not found: C:\...\.ivy2\localorg.apache.spark\spark-core_2.12\3.0.0-preview2\ivys\ivy.xml
    [error]   checksum format error: C:\....\AppData\Local\Coursier\Cache\v1\https\repo1.maven.org\maven2\org\apache\spark\spark-core_2.12\3.0.0-preview2\.spark-core_2.12-3.0.0-preview2.pom__sha1
    [error] Error downloading org.apache.spark:spark-sql_2.12:3.0.0-preview2
    [error]   Not found
    [error]   Not found
    [error]   not found: C:\...\.ivy2\localorg.apache.spark\spark-sql_2.12\3.0.0-preview2\ivys\ivy.xml
    [error]   checksum format error: C:\....\AppData\Local\Coursier\Cache\v1\https\repo1.maven.org\maven2\org\apache\spark\spark-sql_2.12\3.0.0-preview2\.spark-sql_2.12-3.0.0-preview2.pom__sha1
    [error] Total time: 1 s, completed 17 Sep 2022, 15:13:49
    [info] shutting down sbt server

build.sbt is:

/*ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.13.8"

lazy val root = (project in file("."))
  .settings(
    name := "spark-learning"
  )*/

// Name of the package
name := "spark-learning"
// Version of our package
version := "1.0"
// Version of Scala
scalaVersion := "2.12.14"
// Spark library dependencies
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "3.0.0-preview2",
  "org.apache.spark" %% "spark-sql" % "3.0.0-preview2"
)

What causes these issues out of sudden?And how can I get rid?

intellij-idea apache-spark scala
1 Answers

One weird thing in the error messages is the path where it looks for dependencies: ...ivy2\localorg.apache.spark\....

There should be a \ after local.

Any chance your SBT repositories configuration could be messed up? Not sure where it is on Windows, it's /etc/sbt/repositories typically on Linux but maybe Intellij has its own setting.

Gaël J0
Related