Getting error on running ***pyspark --conf spark.jars.packages=com.amazon.deequ:deequ:1.0.2*** in apache spark

user_13385253 0
Viewed 626

I have installed apache-spark 2.4.4 version on ubuntu:16.04 and all it's dependencies. After installation I am running below mentioned pyspark command

pyspark --conf spark.jars.packages=com.amazon.deequ:deequ:1.0.2

after running this I am getting a file missing error in this dir - /home/username/.ivy2/

java.io.FileNotFoundException: File file:/home/streamflux/.ivy2/jars/net.sourceforge.f2j_arpack_combined_all-0.1.jar does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
        at org.apache.spark.SparkContext.addFile(SparkContext.scala:1544)
        at org.apache.spark.SparkContext.addFile(SparkContext.scala:1508)
        at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:462)
        at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:462)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:462)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:238)
        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)
20/05/23 09:43:14 INFO server.AbstractConnector: Stopped Spark@bb24d4c{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
20/05/23 09:43:14 INFO ui.SparkUI: Stopped Spark web UI at http://0babaa1d999c:4040
20/05/23 09:43:14 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/05/23 09:43:14 INFO memory.MemoryStore: MemoryStore cleared
20/05/23 09:43:14 INFO storage.BlockManager: BlockManager stopped
20/05/23 09:43:14 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
20/05/23 09:43:14 WARN metrics.MetricsSystem: Stopping a MetricsSystem that is not running
20/05/23 09:43:14 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/05/23 09:43:14 INFO spark.SparkContext: Successfully stopped SparkContext

I am tried with removing the cache. one thing that I have noticed that there is a file with a similar name - net.sourceforge.f2j_arpack_combined_all-0.1-javadoc.jar

Please help me with this error

pyspark apache-spark java
2 Answers

I ran into a similar issue while running spark-submit with com.amazon.deequ:deequ:1.0.4 the problem can be solved by excluding the package net.sourceforge.f2j_arpack_combined_all-0.1.jar, which is a FORTRAN 2 Java, package if it is not required. You can use the below code for it :

val spark = SparkSession.builder
      .appName("deequ-test")
      .config("spark.jars.packages", "com.amazon.deequ:deequ:1.0.4")
      .config("spark.jars.excludes", "net.sourceforge.f2j:arpack_combined_all")
      .getOrCreate()
user_137895550

Just encountered this while testing a job.

There's a JIRA issue that mentions this happens on similar calls and suggests dependency resolution is affected.

Seems like it wasn't fixed as the original version was considered EOL when it resolved.
However, it still surfaces on 2.4.4 / 2.4.5 as far as I can see ( we're on 2.4.5 )

In our case, as we usually add dependencies directly to our docker builds via sbt, all jars are available and loaded locally.

We just explicitly added the affected package

libraryDependencies += "com.github.fommil.netlib" % "all" % "1.1.2" pomOnly ()

You can either use a similar approach or add any relevant jars directly to $SPARK_HOME/jars or the ivy folder and it should work.

user_13522440
Related