I have incremental raw data that is stored on an Amazon S3 bucket.
I would like to run a Spark job on this raw data to perform some transformations, but as it's a large volume of data, I would prefer to avoid running each time the Spark job from scratch.
Is it possible to persist the state of the Spark job, so that each time I run a new job, I only process the new data that has been added after the previous job?
Thanks a lot