Is there an option in Glue to update a table?

user_5387861 0
Viewed 3405

I have data partitioned by day stored in S3, i.e. customer/year=2020/month=04/day=05, and I have a crawler cataloging that data. Data arrives daily. Is there an option in Glue to update the customer table in that example? For instance, let's say that new customers are discovered on day=06, then, it got added to the table, but let's say that existing customers have updated fields, then, is there an option to only update the table? Or is it a new record to the table?

Currently, when configuring a crawler to discover partitioned data, the partition fields get added to the record. I guess what I'd like to know is if it's possible to constantly have a table representing the latest state of the data?

Thanks in advance. K

aws-glue-data-catalog aws-glue
1 Answers

Summarising your question

Is there an option in Glue to update the customer table in that example?

You can only add new partitions to the table when they come in. This means that if you have a customer record in the previous partitions and the a new record came in day=06 for the same customer, this new record comes in with a new partition date and is visible along with the data for the same customer with previous partitioned dates. You cannot update the same customer record (if its on s3), unless and until you read all the files and do some sort of ranking to create a new dataset.

user_129503400
Related