df.writeStream
.foreachBatch((batchDF: DataFrame, batchId: Long) =>
batchDF.write
.format("org.apache.spark.sql.cassandra")
.cassandraFormat(cassandraTable, cassandraKeyspace, cassandraCluster)
.mode("append")
.save())
.option("checkpointLocation", checkpointDir)
.start()
.awaitTermination()
At the end of the code, dataframe is written to cassandra table.
After checking the last stage through the UI, there was no part to execute save/append-data.
I wonder why it doesn't exist or is there something I've missed.
=========================== After changing my code ===========================
.writeStream
// .foreachBatch((batchDF: DataFrame, batchId: Long) =>
// batchDF.write
// .format("org.apache.spark.sql.cassandra")
// .cassandraFormat(cassandraTable, cassandraKeyspace, cassandraCluster)
// .mode("append")
// .save())
.cassandraFormat(cassandraTable, cassandraKeyspace, cassandraCluster)
.option("checkpointLocation", checkpointDir)
.start()
.awaitTermination()
But I can see WriteToDataSourceV2 in SQL tab.


