Can I control request timeouts within Azure Storage Retry Policies?

Question

It took me some time to understand the idea behind the RetryPolicies the Azure Storage SDK (and probably otehr SDKs as well) provides.

For Microsoft documentation see https://docs.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific#azure-storage-retry-guidelines

Consider the following code:

// [...] get storage, queueclient, queue, ...
CloudQueueMessage message = new CloudQueueMessage ("sample");

OperationContext operationContext = new OperationContext();
operationContext.SendingRequest += (sender, eventArgs) => {
    Console.WriteLine(">> {0}: SendingRequest...", DateTime.Now.ToString("hh:mm:ss.fff tt"));
};

operationContext.Retrying += (sender, eventArgs) => {
    Console.WriteLine(">> {0}: Retrying...", DateTime.Now.ToString("hh:mm:ss.fff tt"));
};

operationContext.RequestCompleted += (sender, eventArgs) => {
    Console.WriteLine(">> {0}: RequestCompleted (took: {1}).",
        DateTime.Now.ToString("hh:mm:ss.fff tt"),
        (eventArgs.RequestInformation.EndTime - eventArgs.RequestInformation.StartTime).TotalMilliseconds);
};

operationContext.ResponseReceived += (sender, eventArgs) => {
    Console.WriteLine(">> {0}: ResponseReceived...", DateTime.Now.ToString("hh:mm:ss.fff tt"));
};

QueueRequestOptions queueRequestOptions = new QueueRequestOptions
{
    // RETRY POLICY
    RetryPolicy = new LinearRetry(TimeSpan.FromMilliseconds(2000), 3),
    MaximumExecutionTime = TimeSpan.FromSeconds(40)
};

try
{
    await queue.AddMessageAsync(message, TimeSpan.FromDays(7), TimeSpan.Zero, queueRequestOptions, operationContext);
}
catch (Exception ex)
{
    Console.WriteLine("EXCEPTION: " + ex.Message);
    return false;
}

Test Setup - I set a local DNS entry to redirect request to messagebox.queue.core.windows.net to a local IP address (which is not responding) in order to simulate connection timeouts.

Observations:

06:13:01.952 : SendingRequest...
06:13:23.036 : RequestCompleted (took: 21083,6253).
06:13:25.040 : Retrying...
06:13:25.040 : SendingRequest...
06:13:41.908 : RequestCompleted (took: 16867,7884).
EXCEPTION: The client could not finish the operation within specified timeout.

The MaximumExecutionTime is a hard time cap in which the client has to sucessfully deliver its message
But why is there only one retry? And why did the request take so long?

The RetryPolicy defines the backing off time.
Looking at the timestamps above you see that the retry ~2000ms after the initial request failed. That's exactly the 2000ms I defined in the RetryPolicy earlier.

What I can unfortunately NOT control is how long a client is trying to deliver its message. If the MaximumExecutionTime is not specified it defaults to 120s.

Results after removing the MaximumExecutionTime property:

06:21:21.681 : SendingRequest...
06:21:42.752 : RequestCompleted (took: 21071,3961).
06:21:44.762 : Retrying...
06:21:44.762 : SendingRequest...
06:22:26.820 : RequestCompleted (took: 42057,7503).
06:22:28.835 : Retrying...
06:22:28.835 : SendingRequest...
06:23:10.868 : RequestCompleted (took: 42033,8183).
06:23:12.886 : Retrying...
06:23:12.886 : SendingRequest...
06:23:54.928 : RequestCompleted (took: 42042,4322).
EXCEPTION: An error occurred while sending the request.

Here are my questions about this...

Ok, so why is the timeout value for the first request ~20s and for all subsequent requests ~40s?
Can I control this? I want requests to fail much faster than 20 oder even 40 seconds!

As a "workaround" I set the MaximumExecutionTime to 500ms.
I set the retry policy to NoRetry.
I am using Polly to handle retries.

Can I control request timeouts within Azure Storage Retry Policies?

0 Answers