Writing data to Cassandra table from Kafka topic failing

user_19820351 0
Viewed 36

I have written simple job which consumes the data from Kafka topic and writes it to cassandra table . I can see data being printed to Console however when it comes to writing it to Cassandra job fails with -

java.lang.NoSuchMethodError: scala.collection.JavaConverters$.mapAsScalaMapConverter

I am using Kafka 3.2.1 , Cassandra 3.11.13 ,pyspark 3.3.0.

my spark-submit command -

spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.13:3.3.0,com.datastax.spark:spark-cassandra-connector_2.12:3.2.0,org.apache.kafka:kafka-clients:3.2.1 --jars /home/hdoop/jars/jsr166e-1.1.0.jar iwd_dproc_new.py
spark-cassandra-connector pyspark apache-spark cassandra
1 Answers

You are using connectors compiled for different Scala versions:

  • Kafka connector is compiled with Scala 2.13
  • Cassandra connector is compiled with Scala 2.12

It's not clear what version of Scala is used for Spark, but you need to take version that uses Scala 2.12, because there is no Spark Cassandra Connector for Scala 2.13. And adjust Kafka connector version as well

Alex_Ott_186270
Related