Batch message processing from Azure Storage Queue to Function App

Gie Grajo 0
Viewed 40

What I want to know is if I can grab, let's say, 1000 messages from Storage queue and process them in single Azure Function run.

I am building a system that will merge the messages data of, let's say, 1000 messages to a json format and send it somewhere. However, the messages are arriving a hundred thousand per minute.

I am thinking to put those messages first in a queue and grab them per batch and the merge the data of each batch.

To consider for the possible solution:

  1. A batch must be processed within a second or less. This is to be used for a real-time application. So, processing of the messages should be as quick as possible.

Added

Another solution comes to my mind for my system is to save these messages to folders separated by time (per second timestamp). Then, the function will just combine the data from each folder.

My question here are:

  1. Is this possible?
  2. Is this a time-effective solution?
  3. What resources are better to use for this solution? ==-

If you have other solutions like using other Azure resources (Note: I am a newbie in Azure world and not familiar of what resources is better cause there are so many). This will be highly appreciated.

azure-storage-queues azure-functions azure
1 Answers

With a single Azure Function it is not possible to do so simply because the maximum number of messages that can be fetched in a single batch/request from an Azure Storage Queue is 32.

From this link:

enter image description here

What you could possibly do is make use of Azure Durable Functions and implement Fan-In Pattern where each instance of your Function will fetch 32 messages from the queue and then send it to an aggregator Function which will merge data from each individual Function fetching messages from the Storage Queue.

Gaurav Mantri0
Related