spark streaming writing entire data instead incremental

Viewed 121

I am trying to write incremental data with below example- source path - s3a://bucketName/inPath with append mode.

I am reading it as stream and writing to currentTimestamp folder each time stream is written as below-

spark.readStream.schema(spark.read.load("s3a://bucketName/inPath").schema).parquet("s3a://bucketName/inPath")
        .writeStream
        .trigger(Trigger.Once)
        .option("checkpointLocation", "s3a://bucektName/outPath/checkpoint")
        .foreachBatch { (batchDF: DataFrame, batchId: Long) =>
            batchDF
            .write
            .mode(SaveMode.Append)
            .parquet(s"s3a://bucektName/outPath/${currentTimestamp}")
        }.start()
     

but its creating full data write to each currentTimestamp folder.

Please help.

0 Answers
Related