I don't understand why there isn't any save or append data operation at the last stage

Viewed 91
      df.writeStream
      .foreachBatch((batchDF: DataFrame, batchId: Long) =>
        batchDF.write
          .format("org.apache.spark.sql.cassandra")
          .cassandraFormat(cassandraTable, cassandraKeyspace, cassandraCluster)
          .mode("append")
          .save())
      .option("checkpointLocation", checkpointDir)
      .start()
      .awaitTermination()

At the end of the code, dataframe is written to cassandra table.

After checking the last stage through the UI, there was no part to execute save/append-data.

enter image description here

I wonder why it doesn't exist or is there something I've missed.

=========================== After changing my code ===========================

      .writeStream
//      .foreachBatch((batchDF: DataFrame, batchId: Long) =>
//        batchDF.write
//          .format("org.apache.spark.sql.cassandra")
//          .cassandraFormat(cassandraTable, cassandraKeyspace, cassandraCluster)
//          .mode("append")
//          .save())
      .cassandraFormat(cassandraTable, cassandraKeyspace, cassandraCluster)
      .option("checkpointLocation", checkpointDir)
      .start()
      .awaitTermination()

enter image description here

But I can see WriteToDataSourceV2 in SQL tab.

enter image description here

1 Answers
Related