Here is my architecture:
I have a blob storage with an event that triggers an azure function when a blob is created in a container.
The azure function read the file (xml) and sent a message to a service bus queue depending on a xml tag in the file. It means that I have multiple queues in my service bus.
An app that is running with multiple threads is then pulling the messages from the queue and execute an action depending on the message.
My problem is that when a massive amount of file is uploaded to the blob (~25 000), then the azure function open a lot of connection (ServiceBusClient) to send the messages to a service bus queue which cause the service bus client to throw an error because I reach the limit of connection (5000 max). It means that my azure function have more than 5000 connections in parallel to the service bus.
Is there a way to limit the azure function to 2000 instances or to slow down the event send to the azure function ?
I already tried to make a retry policy if it fails inside the azure function and in the ServiceBusClient but i would really like to avoid this.
Thanks !