Pyspark cassandra connector NoclassDefFoundError, util/logging

user_9904447 0
Viewed 65

I downloaded the spark-cassandra-connector_3.12-3.2.0.jar and put in /apache-spark/3.2.1/libexec/jars folders.

Calling this jar in conf like:

conf=SparkConf().setAppName("d").setMasetMaster("local")
   .set("spark.driver.extraClassPath",
        "jarPath/spark-cassandra-connector_3.12-3.2.0.jar")

Querying caasandra but getting below error, please suggest

**Py4j.protocol.PyJJavaError: An error occurred while calling o114.load.
:Java.lang.NoClassDefFoundError: com/datastax/spark/connector/util/logging**
spark-cassandra-connector apache-spark cassandra
1 Answers

The problem is that this jar doesn't contain all necessary dependencies. Second problem is that you're using incorrect configuration property.

To fix it you have two choices:

  1. You use spark.jars.packages configuration property (or --packages in the command-line) with value com.datastax.spark:spark-cassandra-connector_2.12:3.2.0 as described in the docs. This will fetch the jar + necessary dependencies

  2. You instead download the spark-cassandra-connector-assembly jar that includes all dependencies and use it with spark.jars configuration property (or --jars in the command-line).

P.S. See Spark docs about configuration properties.

Alex_Ott_186270
Related