I have a few Azure Service Bus listener functions that can be simplified as follows:
public class MyFunction
{
private readonly IMyInsightsLogger logger;
private readonly ISomeBusinessManager businessManager;
private readonly IConfiguration configuration;
public MyFunction(IMyInsightsLogger logger,
ISomeBusinessManager businessManager, IConfiguration configuration)
{
this.logger = logger;
this.businessManager = businessManager;
this.configuration = configuration;
}
[FunctionName("MyFunction")]
public async Task Run([ServiceBusTrigger("my.topic", "my.sub",
Connection = "AzureServiceBusConnectionString", IsSessionsEnabled = true)]
ServiceBusReceivedMessage receivedMessage,
ServiceBusMessageActions messageReceiver,
CancellationToken cancellationToken)
{
// some code for testing purposes to check if the problem is caused by initially cancelled token
if (cancellationToken.IsCancellationRequested)
{
logger.Error("P1 cancellation token already expired");
}
else
{
logger.Information("P1 cancellation token OK");
}
try
{
await businessManager.HandleMyMessage(receivedMessage);
// some code for testing purposes to check if the problem is caused by a cancelled token after we did some processing of the message
if (cancellationToken.IsCancellationRequested)
{
logger.Error("P2 cancellation token already expired");
}
else
{
logger.Information("P2 cancellation token OK");
}
// the culprit of the problem - manual completion
await messageReceiver.CompleteMessageAsync(receivedMessage, cancellationToken);
logger.Information("P3 Done OK");
}
catch (Exception e)
{
logger.Error(e, $"Error when processing a message of type {receivedMessage.ContentType} \nException details: {e.Message}");
// rethrow to increase DeliveryCount and make the message go to deadletters
throw;
}
}
}
Host.json looks like this:
{
"version": "2.0",
"extensions": {
"serviceBus": {
"prefetchCount": 0,
"autoCompleteMessages": false,
"maxConcurrentSessions": 30
}
}
}
I'm intentionally using autoCompleteMessages: false because some of the functions need more complex workflow to decide when the message should be left in the queue and when I can call CompleteMessageAsync explicitly.
It all seems to work nice with a few messages. But when I did some stressful tests with real-life expected scenarios with batches of few thousands of messages, the functions (even a simple empty test function without HandleMyMessage) started to behave strangely. It picked a few dozen messages and then it started failing on line CompleteMessageAsync with the following errors in our logs (using Azure Application Insights):
13:50:06
-
EXCEPTION
A task was canceled.
Problem Id: System.Threading.Tasks.TaskCanceledException at Azure.Messaging.ServiceBus.Core.CancellationTokenExtensions.ThrowIfCancellationRequested
FormattedMessage: ...Id=756391f1-5c75-467b-9f2d-... Duration=11006ms)
13:50:06
-
TRACE
Executed 'MyFunction' (Failed, Id=756391f1-5c75-467b-9f2d-..., Duration=11006ms)
Severity level: Error
What's strange - this problem seems to happen only on Azure. When I run the same function locally consuming the same "stuck" messages, it works just fine eating up a few hundred messages immediately.
After a few hours, the stuck Azure function seems to normalize for a few minutes. Then it processes a few dozen messages normally, and then again it falls into the TaskCanceledException abyss for hours.
My two checks for cancellationToken.IsCancellationRequested are never triggered, so as far as I know, the token is not cancelled before it reaches CompleteMessageAsync.
When I remove the CompleteMessageAsync and "autoCompleteMessages": false everything starts working normally, the function consumes thousands of messages in an hour without any TaskCanceledException.
The functions are the only listeners on their subscriptions, so there are no should be no conflicts. I also tried "maxConcurrentSessions": 1 - it slowed things down a lot, but still ended up with the same issue.
This behavior leads me to think that my intuitive implementation of passing the function cancellationToken down the chain to CompleteMessageAsync is not the right way.
Then what is the right way? What am I supposed to do with cancellationToken and what should I pass to CompleteMessageAsync as cancellationToken? How is Microsoft's autoCompleteMessages able to consume the same messages without failing because of TaskCanceledException?