Re-play/Repeat/Re-Fire Azure BlobStorage Function Triggers for existing files

user_1662268 0
Viewed 1040

I've just uploaded several 10s of GBs of files to Azure CloudStorage. Each file should get picked up and processed by a FunctionApp, in response to a BlobTrigger:

[FunctionName(nameof(ImportDataFile))]
public async Task ImportDataFile(
    // Raw JSON Text file containing data updates in expected schema
    [BlobTrigger("%AzureStorage:DataFileBlobContainer%/{fileName}", Connection = "AzureStorage:ConnectionString")]
    Stream blobStream,
    string fileName)
{
    //...
}

This works in general, but foolishly, I did not do a final test of that Function prior to uploading all the files to our UAT system ... and there was a problem with the uploads :(

The upload took a few days (running over my Domestic internet uplink due to CoViD-19) so I really don't want to have to re-do that.

Is there some way to "replay" the BlobUpload Triggers? so that the function triggers again as if I'd just re-uploaded the files ... without having to transfer any data again!

azure-triggers azure-blob-storage azure-functions azure
4 Answers

As per this link

Azure Functions stores blob receipts in a container named azure-webjobs-hosts in the Azure storage account for your function app (defined by the app setting AzureWebJobsStorage).

To force reprocessing of a blob, delete the blob receipt for that blob from the azure-webjobs-hosts container manually. While reprocessing might not occur immediately, it's guaranteed to occur at a later point in time. To reprocess immediately, the scaninfo blob in azure-webjobs-hosts/blobscaninfo can be updated. Any blobs with a last modified timestamp after the LatestScan property will be scanned again.

user_1880960

I found a hacky-AF work around, that re-processes the existing file:

If you add Metadata to a blob, that appears to re-trigger the BlobStorage Function Trigger.

Accessed in Azure Storage Explorer, but Right-clicking on a Blob > Properties > Add Metadata.

I was settings Key: "ForceRefresh", Value "test".

user_16622680

I had a problem with the processing of blobs in my code which meant that there were a bunch of messages in the webjobs-blobtrigger-poison queue. I had to move them back to azure-webjobs-blobtrigger-name-of-function-app. Removing the blob receipts and adjusting the scaninfo blob did not work without the above step.

Fortunately Azure Storage Explorer has a menu option to move the messages from one queue to another:

enter image description here

user_23010650

I found a workaround, if you aren't invested in the file name:

Azure Storage Explorer, has a "Clone with new name" button in the top bar, which will add a new file (and trigger the Function) without transferring the data via your local machine.

Note that "Copy" followed by "Paste" also re-triggers the blob, but appears to transfer the data down to your machine and then back up again ... incredibly slowly!

user_16622680
Related