I need to have a glue job that appends data post-curation. My workflow right now looks like:
S3 buckets:
- raw/file.csv
- clean
- year=*/
- month=*/
- day=*/
- run-unnamed-part-000000
- curated/curated_file.csv
GlueJobs:
- RawToClean (Bookmark Enabled)
- CleanToCurate (Bookmark Enabled)
My Job scripts read from_catalog and use a transformation_ctx for both jobs.
Let's say I have:
- RawFile1.csv: 100 Rows
- RawFile2.csv: 50 Rows
For example, after running RawFile1.csv through the process, my clean bucket will look like cleaned/year=2022/month=09/day=21/run-unnamed-1-part-00000. My curated bucket will have curated.csv in it, with 100 rows. Now, I delete RawFile1.csv in raw, and drop RawFile2.csv in the bucket and run it through the process. Now the clean Bucket will look like (in the same year/mothn/day/ folder):
- run-unnamed-1-part-00000
- run-unnamed-2-part-00000
I have logic in CleanToCurate that reads curated.csv as a DF from the Curated Bucket, and appends that DF to my DF that was created from_catalog from the Clean Bucket. I want this to output 150 rows, but instead it outputs ~250 rows. When I read from_catalog (from Cleaned) it is reading both run-* files (150 rows total) into the DF and then appending (using PySpark Union) that to the previous curated.csv (100 rows). From what I have read in documentation, bookmarking should know not to grab run-unnamed-1-part-00000 (Cleaned RawFile1.csv, 100 Rows) and only grab run-unnamed-2-part-00000 (Cleaned RawFile2.csv, 50 rows).