Azure Functions ServiceBus Trigger Scaling Behavior

Viewed 532

We are currently running load tests on our Azure Function App but the throughput is not what we expected.

There are multiple functions in the Function App but the ones with the most traffic are one with an Event Hub Trigger and one with a Service Bus Trigger consuming messages from a Session-Enabled Queue.

When the system is under load, Messages in the Session-Enabled Queue wait for up to 10 Minutes in the queue until they get processed by the consuming Function.

I know there are some settings in host.json to tune this behavior but it's still far from what we expect.

This is our host.json

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    }
  },
  "extensions": {
    "serviceBus": {
      "prefetchCount": 100,
      "sessionHandlerOptions": {
        "autoComplete": true,
        "messageWaitTimeout": "00:00:30",
        "maxAutoRenewDuration": "00:55:00",
        "maxConcurrentSessions": 200
      },
      "batchOptions": {
        "maxMessageCount": 1000,
        "operationTimeout": "00:01:00",
        "autoComplete": true
      }
    }
  }
}

So i would expect the Function App to process up to 200 Sessions concurrently but in fact, although the Function Runtime provisions lots of instances, most of them seem to sit there and idle out. So to me it seems there is still another setting limiting the througput of the Function App.

Application Insights Monitoring

I know it would improve performance if we would split the functions to separate Function Apps but as the load on the both functions is quite similar my plan was to postpone this step to a later stage and still get acceptable throughput with a single Function App.

We are using Azure Functions 3 on .NET Core 3.1 with

  • Microsoft.Azure.Functions.Extensions 1.1.0
  • Microsoft.Azure.WebJobs.Extensions.ServiceBus 5.0.0
  • Microsoft.Azure.WebJobs.Extensions.EventHubs 5.0.0

on a Windows Consumption Plan.

Thank you for any hints on how to achieve acceptable throughput.

1 Answers

I figured out that handling Batch Messages (receiving ServiceBusMessage[] instead of single instances) in the ServiceBus-Triggered Function along with enabled Sessions has a massively negative impact on scalability.

After changing this to single instances, the behavior of the system was as expected and the sessionHandlerOptions in host.json were respected.

I am wondering though what's the reason for this. I guess it could be related to the circumstance that Azure Function Instances lease a number of sessions from Service Bus to process but could not find anything in the documentation on that.

Related