I am trying to write incremental data with below example- source path - s3a://bucketName/inPath with append mode.
I am reading it as stream and writing to currentTimestamp folder each time stream is written as below-
spark.readStream.schema(spark.read.load("s3a://bucketName/inPath").schema).parquet("s3a://bucketName/inPath")
.writeStream
.trigger(Trigger.Once)
.option("checkpointLocation", "s3a://bucektName/outPath/checkpoint")
.foreachBatch { (batchDF: DataFrame, batchId: Long) =>
batchDF
.write
.mode(SaveMode.Append)
.parquet(s"s3a://bucektName/outPath/${currentTimestamp}")
}.start()
but its creating full data write to each currentTimestamp folder.
Please help.