We have set up a Azure Function with an eventHubTrigger to read blobs created by Apache Nifi. The EventGridSchema is filtered for Blob Created events, largely working fine. The problem is that the Azure Function occasionally fetches the blob before it is fully committed.
Storage account is of type BlockBlobStorage with Hierarchical namespace. The StorageBlobLogs shows that the typical sequence of operations:
Nifi processor: CreatePathFile > AppendFile > FlushFile.
Followed by the Azure Function doing Getblob > DeleteBlob. When the problem occurs, the GetBlob operation happens prior to FlushFile or even prior to AppendFile
It seems that only the CreatePathFile operation trigges Blob Created events. The docs indicates that FlushWithClose would also trigger it, and we have tried applying an event filter on data.api without achieving anything other than stopping function runs altogether.
How can we setup a subscription for Blob Created Event triggered strictly after the Blob has been fully committed?