Qubole#sparklens;0.3.1-s_2.11: not found (pyspark on EMR)

user_17867413 0
Viewed 76

I am trying to execute sparklens through EMR and getting this error , below is my spark-submit command .

I am using pyspark 3.0.1 and hadoop 3.1.1 verstion

spark-submit --packages Qubole:sparklens:0.3.1-s_2.11 --conf Spark.extraListeners=com.qubole.Sparklens.QuboleJobListener --py-files config.zip,jobs.zip,DDL.zip main.py

I am getting

problems summary :: :::: WARNINGS module not found: qubole#sparklens;0.3.0-s_2.11

==== local-m2-cache: tried

  file:/home/hadoop/.m2/repository/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.pom

  -- artifact qubole#sparklens;0.3.0-s_2.11!sparklens.jar:

  file:/home/hadoop/.m2/repository/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.jar

==== local-ivy-cache: tried

  /home/hadoop/.ivy2/local/qubole/sparklens/0.3.0-s_2.11/ivys/ivy.xml

  -- artifact qubole#sparklens;0.3.0-s_2.11!sparklens.jar:

  /home/hadoop/.ivy2/local/qubole/sparklens/0.3.0-s_2.11/jars/sparklens.jar

==== central: tried

  https://repo1.maven.org/maven2/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.pom

  -- artifact qubole#sparklens;0.3.0-s_2.11!sparklens.jar:

  https://repo1.maven.org/maven2/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.jar

==== spark-packages: tried

  https://dl.bintray.com/spark-packages/maven/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.pom

  -- artifact qubole#sparklens;0.3.0-s_2.11!sparklens.jar:

  https://dl.bintray.com/spark-packages/maven/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.jar

    ::::::::::::::::::::::::::::::::::::::::::::::

    ::          UNRESOLVED DEPENDENCIES         ::

    ::::::::::::::::::::::::::::::::::::::::::::::

    :: qubole#sparklens;0.3.0-s_2.11: not found

    ::::::::::::::::::::::::::::::::::::::::::::::

Thanks, Xi

pyspark apache-spark-sql apache-spark
1 Answers

It looks like the sparklens module cannot be found in any of the Spark repositories.

A solution is to download the jar file sparklens-0.3.0-s_2.11.jar from https://mvnrepository.com/artifact/qubole/sparklens/0.3.0-s_2.11 and use the option --jars in place of --packages like this:

spark-submit --jars ./sparklens-0.3.0-s_2.11.jar ...

(assuming the jar is in the current folder)

Alternatively, you could specify a different repository with the option --repositories but I didn't find a working repository https://repos.spark-packages.org and bintray.com both didn't work)

user_23147370
Related