I’m working on a project to pull data from a csv file that resides in an amazon s3 bucket onto Bigquery using a data transfer. The csv file gets dropped into s3 every day, replacing the previous day’s file under the same name. I set up a data transfer to run every day pulling this data into a table in Bigquery. However, when the transfer runs, instead of overwriting the previous day’s data in the table, it’s simply appending it, thereby duplicating the data. Does anybody have any idea on how to tackle this issue?
0 Answers
Related