I'm a newbie to .net and Azure. I'm trying to design a pipeline to process files created in Azure container. I have a created event grid blob trigger on the container to get the metadata of the files created. I have two options now
Use Azure function to consume the metadata from event grid and process it. I believe the Azure function can scale out based on traffic in event grid. But the files could be large, having a size of 60 GB. I read Azure functions are not ideal for long processing. Does Azure function works for my case ?
Use storage queue to consume the metadata from event grid. Create an application to consume from the queue and process files.
Please suggest what kind of application I can develop and deploy so that I could achieve scale out/in based on the queue traffic and process large blob files efficiently.