I'm working on a project where we need to bring data from SQL Server database into a Datalake. I succeded that through a pipeline which ingest data from the source and load it into a DL in parquet format. My question is how to merge new data from data source to the existing file into that data lake(Upserting).
1 Answers
You can use Azure data flows wherein you can map the source file with other sources and override the existing file. There is no upsert activity directly in ADF for files unlike for databases. reference : https://learn.microsoft.com/en-us/answers/questions/542994/azure-data-factory-merge-2-csv-files-with-differen.html
Related