Azure: Best architecture to run a lot of tasks that takes between 5 and 10 seconds each (triggered from web api)?

user_1944552 0
Viewed 95

We are using Azure services for this, and we have a web api (Flask) running on Azue Web Service App, connected to a PostgreSQL in a different service (so far so good, although the connection is a bit slow). Anyway, there is an action performed by users that triggers a process. Basically, we need to pre-process thousands of records (between 1000 and 5000), and processing each one takes between 5 and 10 seconds.

We thought in using Azure Functions to this, and trigger thousands of http requests. But the requests are not parallelized, even when we added more instances. It seems they are queued. I talked to Azure support and they told me this may not the best solution, since Azure functions are expected to run quick, not take several seconds to perform.

So my question is: what's the best architecture for doing something like this on Azure? I read about something called Event Grid, but I have no clue (first time using Azure). The basic solution would be to have a VM, that can receive parameters and start processes, but that's something that would be too complicated (and we don't have a devops in our team that could take care of that).

Thanks in advance.

azure execution architecture events
3 Answers

I would strongly recommend evaluating azure durable functions for this, the runtime supports fan out automatically meaning you can get massive scale very easily. Make sure the systems these functions talk to can also handle the load!

The basic structure is that you kick off an orchestrator function which can then kick off multiple activity functions, which if necessary can themselves be chained together or even fanned out again. The timings you’ve quoted would not be an issue see here. You can run all this on the consumption plan happily.

Of course you need to look into cost here, if you do use the consumption plan you will pay by the Mb-second, ie cost is proportional to memory used and time taken.

user_9344070

You can try the Azure Queue storage trigger it will trigger the azure function as you add a message in the azure queue and yeah it will run parallel But it has a limit but you can scale it as it's mention in the documentation.

Azure Queue storage trigger for Azure Functions

user_52419740

Matt's answer is will work, however I specifically caught that you would prefer the requests to be parallelized (i.e. time to execute is a priority). Have you looked at Azure Batch ? It is specifically designed for large scale parallel workloads.

Admin1
Related