Redis System.TimeoutException: The operation has timed out

Viewed 48

While investigating some error logs I have noticed that our Redis client is throwing the following exception:

System.TimeoutException: The operation has timed out.
  at StackExchange.Redis.ConnectionMultiplexer.Wait[T](Task`1 task) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 405

The line where this happens as you can see when we do an Async task:

var db = _connector.GetDatabase();
var task = db.HashSetAsync(hashKey
    , Id
    , _GetNewExp(expirationInSeconds)
    , When.Always);
db.Wait(task);

We are only logging this error after retrying 2 times and there is no extra information coming from the exception even if our config is set up to give us details in the exception. The timeout is matching the SyncTimeout set in the options.

_config = new ConfigurationOptions
          {
             AbortOnConnectFail = true,
             ConnectRetry = 2,
             ConnectTimeout = 1300,
             SyncTimeout = 1600,
             KeepAlive = 180,
             ConfigCheckSeconds = 15,
             IncludeDetailInExceptions = true,
             EndPoints =
             {
                 { host, port}
             },
             ResponseTimeout = 2000
         };

The lua script we are running on redis has a hard cap of 25 HGET/HSET, so as far as we are aware we are not deadlocking ourselves in that end.

While inspecting on redis using RedisInsight we noticed that when these errors happens we have a drop of commands followed by a spike in them, this is leading me to believe that it might be a slowdown in AWS, but we don't have any conclusive evidence about this.

Spike seen in Redis Insight:

1

If anyone can provide any insight that could help us prevent this it would help a lot.

Redis version is 6.2.5 and the StackExchange.Redis library version is 2.5.61

0 Answers
Related