We are in the process of implementing auto ingestion of data between a snowflake data warehouse and an azure data container (enter link description here)
Based on the snowflake documentation we created an Azure storage account, a container where the blobs will be downloaded nightly, a queue and added the event subscription whose end point is the storage queue that filters on the blob created event. Below is a screen shot of the Azure event subscription.
We use a logic app to connect to the client FTP site to download the files to the storage container. A simple FTP job - Trigger when a new file is added, Get File content using path and Create blob. Screen shot below
However what we have been seeing is that for certain files the data ingestion starts before the blob is committed to the storage container. This results in a failure to load data. After working with support at snowflake and trouble shooting at our end it does seem like the blob created event is fired before the blob is committed.
Is there any way for me to resolve this. Or any pointers as to how to investigate this further.
Regards Sid
