Improve performance of Redis to reduce timeout exceptions

Viewed 450

I have a Redis Database on a Centos server, and 3 Windows servers are connected to it with approximately 1,000 reads/writes per second, all of which are on the same local LAN, so the ping time is less than one millisecond. The problem is at least 5 percent of reading operations are going timeout, while I read maximum 3KB data in a read operation with 'syncTimeout=15', which is much more than network latency.

I installed Redis on bash on my windows 10, and simulate the problem. I also stopped writing operations. However, the problem still exists with 0.5 percent timeouts, while there is no network latency. I also used a Centos Server in my LAN to simulate the problem, in this case, I need at 100 milliseconds for 'syncTimeout' to be sure the amount of timeout is less than 1 percent. I considered using some Dictionaries to cache data from Redis, so there is no need to request per item, and I can take advantage of the pipeline. But I came across StackRedis.L1 which is developed as an L1 cache for Redis, and it is not confident in updating the L1 cache.

This is my code to simulate the problem:

    var connectionMulti = ConnectionMultiplexer.Connect(
            "127.0.0.1:6379,127.0.0.1:6380,allowAdmin=true,syncTimeout=15");

    // 100,000 keys
    var testKeys = File.ReadAllLines("D:\\RedisTestKeys.txt");

    for (var i = 0; i < 3; i++)
    {
        var safeI = i;
        Task.Factory.StartNew(() =>
        {
            var serverName = $"server {safeI + 1}";
            var stringDatabase = connectionMulti.GetDatabase(12);
            PerformanceTest($"{serverName} -> String: ",
                key => stringDatabase.StringGet(key), testKeys);
        });
    }

and the PerformanceTest method is:

private static void PerformanceTest(string testName, Func<string, RedisValue> valueExtractor,
        IList<string> keys)
    {
        Task.Factory.StartNew(() =>
        {
            Console.WriteLine($"Starting {testName} ...");
            var timeouts = 0;
            var errors = 0;
            long totalElapsedMilliseconds = 0;

            var stopwatch = new Stopwatch();
            foreach (var key in keys)
            {
                var redisValue = new RedisValue();
                stopwatch.Restart();
                try
                {
                    redisValue = valueExtractor(key);

                }
                catch (Exception e)
                {
                    if (e is TimeoutException)
                        timeouts++;
                    else
                        errors++;
                }
                finally
                {
                    stopwatch.Stop();
                    totalElapsedMilliseconds += stopwatch.ElapsedMilliseconds;
                    lock (FileLocker)
                    {
                        File.AppendAllLines("D:\\TestResult.csv",
                            new[]
                            {
                                $"{stopwatch.ElapsedMilliseconds.ToString()},{redisValue.Length()},{key}"
                            });
                    }
                }
            }

            Console.WriteLine(
                $"{testName} {totalElapsedMilliseconds * 1.0 / keys.Count} (errors: {errors}), (timeouts: {timeouts})");
        });
    }

I expect all read operations will be done successfully less than 15 milliseconds. Achieving this, is Considering L1 cache for a Redis cache a good solution? (It is very fast, in the scale of a nanosecond, but how can I do for syncronizing) Or Redis can be enhanced by clustering or something else? (While I tested it on bash on my PC, and I did not receive expected result)

1 Answers

Or Redis can be enhanced by clustering or something else?

Redis can be clustered, in different ways:

  • "regular" redis can be replicated to secondary read-only nodes, on the same machine or different machines; you can then send "read" traffic to some of the replicas
  • redis "cluster" exists, which allows you to split (shard) the keyspace over multiple primaries, sending appropriate requests to each node
  • redis "cluster" can also make use of readonly replicas of the sharded nodes

Whether that is appropriate or useful is contextual and needs local knowledge and testing.

Achieving this, is Considering L1 cache for a Redis cache a good solution?

Yes, it is a good solution. A request you don't make is much faster (and has much less impact on the impact) than a request you do make. There are tools for helping with cache invalidation, including using the pub/sub API for invalidations. Redis vNext is also looking into additional knowledge APIs specifically for this kind of L1 scenario.

Related