I was trying to use avro support for spark-2.4.0, as mentioned here. However, since I'm behind a proxy, I wasn't able to download through maven.
Instead, I downloaded the jar to local, launched spark-shell and then inside the spark-shell, typed:
:require /path/to/newly/downloaded/jar.
This throws the error:
The path '/Users/hay531/Desktop/jars/spark-avro_2.11-2.4.0.jar' cannot be loaded, because existing classpath entries conflict.
So, I printed out all the jars in the classpath using:
import java.lang.ClassLoader
val cl = ClassLoader.getSystemClassLoader
cl.asInstanceOf[java.net.URLClassLoader].getURLs.foreach(println)
And below is the list of jars I got:
file:/Users/hay531/Downloads/spark-2.4.0/conf/
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/breeze-macros_2.11-0.13.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/stream-2.7.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/machinist_2.11-0.6.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jersey-media-jaxb-2.22.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-unsafe_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-common-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-shuffle-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/minlog-1.3.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jersey-client-2.22.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/xz-1.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/pyrolite-4.13.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jsr305-1.3.9.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-yarn-common-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/curator-recipes-2.6.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-configuration-1.6.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-beanutils-1.7.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/RoaringBitmap-0.5.11.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/httpcore-4.4.10.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-streaming_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/objenesis-2.5.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-httpclient-3.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/stax-api-1.0-2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hk2-api-2.4.0-b34.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/apacheds-i18n-2.0.0-M15.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/janino-3.0.9.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jtransforms-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/activation-1.1.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/orc-mapreduce-1.5.2-nohive.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jaxb-api-2.2.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/oro-2.0.8.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jackson-jaxrs-1.9.13.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/xercesImpl-2.9.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/scala-xml_2.11-1.0.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/scala-library-2.11.12.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jersey-guava-2.22.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/opencsv-2.3.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-graphx_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-compiler-3.0.9.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/json4s-ast_2.11-3.5.3.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/javax.servlet-api-3.1.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-hdfs-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/httpclient-4.5.6.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jackson-core-2.6.7.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/log4j-1.2.17.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-yarn-server-common-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/arpack_combined_all-0.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/parquet-jackson-1.10.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-cli-1.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/scala-parser-combinators_2.11-1.1.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/validation-api-1.1.0.Final.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/json4s-core_2.11-3.5.3.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-digester-1.8.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-jobclient-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-annotations-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/protobuf-java-2.5.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/javassist-3.18.1-GA.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-auth-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/javax.annotation-api-1.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/xmlenc-0.52.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jackson-xc-1.9.13.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jul-to-slf4j-1.7.16.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jetty-util-6.1.26.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jackson-module-scala_2.11-2.6.7.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/arrow-vector-0.10.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/orc-shims-1.5.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-sketch_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-yarn-client-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jersey-container-servlet-2.22.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/metrics-graphite-3.1.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/paranamer-2.8.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/orc-core-1.5.2-nohive.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/chill_2.11-0.9.3.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-mllib_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/parquet-format-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/htrace-core-3.0.4.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-compress-1.8.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/macro-compat_2.11-1.1.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/chill-java-0.9.3.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/avro-ipc-1.8.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/parquet-common-1.10.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/univocity-parsers-2.7.3.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-math3-3.4.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-io-2.4.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/parquet-hadoop-1.10.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/netty-all-4.1.17.Final.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-app-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jackson-core-asl-1.9.13.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/breeze_2.11-0.13.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/osgi-resource-locator-1.0.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/core-1.1.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/xbean-asm6-shaded-4.8.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/py4j-0.10.7.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-catalyst_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/shapeless_2.11-2.3.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/curator-framework-2.6.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-codec-1.10.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/guava-14.0.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-beanutils-core-1.8.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/snappy-java-1.1.7.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-kvstore_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hppc-0.7.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/aopalliance-repackaged-2.4.0-b34.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/curator-client-2.6.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spire_2.11-0.13.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/zookeeper-3.4.6.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-client-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/arrow-memory-0.10.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/avro-mapred-1.8.2-hadoop2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-collections-3.2.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/api-asn1-api-1.0.0-M20.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/apacheds-kerberos-codec-2.0.0-M15.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jackson-databind-2.6.7.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jackson-module-paranamer-2.7.9.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hk2-locator-2.4.0-b34.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/parquet-column-1.10.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jcl-over-slf4j-1.7.16.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/scala-compiler-2.11.12.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-crypto-1.0.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jersey-container-servlet-core-2.22.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/avro-1.8.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/api-util-1.0.0-M20.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/compress-lzf-1.0.3.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/javax.ws.rs-api-2.0.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/metrics-core-3.1.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-common-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-net-3.1.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/gson-2.2.4.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-launcher_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/ivy-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/javax.inject-2.4.0-b34.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/metrics-json-3.1.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-lang-2.6.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/lz4-java-1.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/antlr4-runtime-4.7.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-repl_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/slf4j-api-1.7.16.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/parquet-encoding-1.10.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/zstd-jni-1.3.2-2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/leveldbjni-all-1.8.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-core-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-tags_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hk2-utils-2.4.0-b34.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/arrow-format-0.10.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/scala-reflect-2.11.12.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-sql_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spire-macros_2.11-0.13.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-mllib-local_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jackson-mapper-asl-1.9.13.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/flatbuffers-1.2.0-3f79e055.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/kryo-shaded-4.0.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jersey-common-2.22.2.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/aircompressor-0.10.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/netty-3.9.9.Final.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/slf4j-log4j12-1.7.16.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/metrics-jvm-3.1.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-network-common_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-core_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/hadoop-yarn-api-2.6.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jackson-annotations-2.6.7.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/json4s-jackson_2.11-3.5.3.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/spark-network-shuffle_2.11-2.4.0.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/joda-time-2.9.3.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/commons-lang3-3.5.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/json4s-scalap_2.11-3.5.3.jar
file:/Users/hay531/Downloads/spark-2.4.0/assembly/target/scala-2.11/jars/jersey-server-2.22.2.jar
However, there was no conflicting jar name in the list. So:
- Why is it that I'm getting an existing classpath entries conflict?
- How can I resolve 1?
- In case there actually are existing conflicts, how do I force
scala/sparkto load the jar which I specify on the CLI?