java.lang.NoClassDefFoundError: com/datastax/spark/connector/ColumnSelector while building a jar

user_11251644 0
Viewed 71

Am using maven project with spark, and a cassandra as a database. Am able to execute the code with 0 error, and getting the result that I want, but I can't submit the project with the jar file: here is my pom.xml :

POM.xml

my commands to submit the project:

spark-submit --master "local[*]" --class com.sparkfinal.App target/sparkfinal-1.0-SNAPSHOT.jar

the error:

ERROR

spark-cassandra-connector datastax apache-spark maven cassandra
1 Answers

You need either:

  1. create an assembly jar for your project that will include all dependencies (make sure you declare Spark dependencies as provided)
  2. or specify Spark Cassandra Connector as dependency when submitting the job by using --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0 as shown in the connector's docs

I would recommend to use first method because I see that you have other dependencies that you'll need to provide. Also, please note that you may not need to specify java driver dependency as it will be pulled by Spark Cassandra Connector automatically.

Alex_Ott_186270
Related