WebJob QueueTrigger TimeoutException: The operation 'GetMessages' did not complete in '00:02:00'

user_265165 0
Viewed 870

A bit related to my issue 60304305... My Azure webJob application handles multiple queues, but they don't generate high traffic. Still, the app will encounter a Timeout Exception from time to time, and it's almost impossible to reproduce the error, but I had some success when breaking at a Breakpoint inside a Queue function and leave it for a while.

System.TimeoutException
  HResult=0x********
  Message=The operation 'GetMessages' with id '********-****-****-****-************' did not complete in '00:02:00'.
  Source=Microsoft.Azure.WebJobs.Extensions.Storage
  StackTrace:
   at Microsoft.Azure.WebJobs.Extensions.Storage.TimeoutHandler.<ExecuteWithTimeout>d__1`1.MoveNext() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\TimeoutHandler.cs:line 30
   at Microsoft.Azure.WebJobs.Host.Timers.WebJobsExceptionHandler.<>c__DisplayClass3_0.<OnUnhandledExceptionAsync>b__0() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Timers\WebJobsExceptionHandler.cs:line 54
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

  This exception was originally thrown at this call stack:
    Microsoft.Azure.WebJobs.Extensions.Storage.TimeoutHandler.ExecuteWithTimeout<T>(string, string, Microsoft.Azure.WebJobs.Host.Timers.IWebJobsExceptionHandler, System.Func<System.Threading.Tasks.Task<T>>) in TimeoutHandler.cs
    Microsoft.Azure.WebJobs.Host.Timers.WebJobsExceptionHandler.OnUnhandledExceptionAsync.AnonymousMethod__0() in WebJobsExceptionHandler.cs
    System.Threading.ThreadHelper.ThreadStart_Context(object)
    System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, object, bool)
    System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, object, bool)
    System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, object)
    System.Threading.ThreadHelper.ThreadStart()

Interestingly, the namespace appears to be "Microsoft.Azure.WebJobs.Host.Timers" but I'm not using any timers, only queues. Also, the Timeout of '00:02:00' is long. And the functions could run for hours but then fail at some time. The big problems here are:

  1. The exceptions appears "out of nowhere", it cannot be catched or gracefully handled, or at least I don't know how, so any help is appreciated
  2. No further information available, I'd like to know which Queue (is it always the same? etc.)

I tried updating the ServicePointManager.DefaultConnectionLimit to a high number as advised here with no success, tried multiple different Queue settings (BatchSize, MaxDequeueCount etc. going from low to high numbers, also no success etc.) I'm tired of trying things, I need to find more information but I don't know how.

Versions:

  • Microsoft.Azure.WebJobs 3.0.14
  • Microsoft.Azure.WebJobs.Extensions 3.0.6
  • Microsoft.Azure.WebJobs.Extensions.Storage 3.0.10
  • Microsoft.Azure.WebJobs.Host.Storage 3.0.14
azure-webjobs azure .net
1 Answers

I also have those weird errors, and I don't know where they come from either. However, I found a way to handle them: when the host is shutting down, the CancellationToken is cancelled and you can monitor its cancellation via the Register method.

For example:

[FunctionName("MyFunction")]
public async Task ProcessMyQueueMessageAsync(
    [QueueTrigger("myqueue")] MyQueueMessage payload,
    ExecutionContext context, CancellationToken cancellationToken)
{
    token.Register(async () =>
    {
        // your code to handle such an error, can be an async function
    });
}
user_26981190
Related