Reading a large number of keys in high Redis traffic

Viewed 26

I have used Redis to store the statistics of a website with relatively high traffic. I get a Redis variable for each statistical item and increment it with each call. Now, I need to read this information every minute and update it in SQL and reset the keys to zero. Since the number of keys and traffic is high, what is the optimal way to read the keys?

The easiest method is to scan the Redis database every minute. But since scanning is costly and it will include new keys that are alive for less than a minute, this method does not seem suitable. In another method, I first enabled the keyspace events to receive a notification when the keys expires.

CONFIG SET notify-keyspace-events Ex

Then, for each key, I created an empty key with a TTL of one minute, and at the time of receiving its expiration notification, I read the associated key and its value and put it in the output queue. This method also has disadvantages, such as the possibility of losing notifications when reconnecting.

public void SubscribeOnStatKeyExpire()
{
    sub.Subscribe("__keyevent@0__:expired").OnMessage(async channelMessage => {
        await ExportExpiredKeys((string)channelMessage.Message);
    });            
}

private async Task ExportExpiredKeys(string expiredKey)
{
    if(expiredKey.StartsWith(_notifyKeyPrefix))
    {
        var key = expiredKey.Remove(0, _notifyKeyPrefix.Length);                
        var value = await _cache.StringGetDeleteAsync(key);
        if (!value.HasValue || value.IsNullOrEmpty) return;
        var count = (int)value;
        if(count <= _negligibleValue)
        {
            await _cache.StringIncrementAsync(key, count);
            return;
        }
        var listLength = await Enqueue(_exportQueue, key + "." + value);
        if (listLength > _numberOfKeysInOneMessage) // _numberOfKeysInOneMessage for now is 500
        {
        var list = await Dequeue(_exportQueue,_numberOfKeysInOneMessage);
                await SendToSqlServiceBrocker(list);
        }
    }            
}

Is there a better way? For example, using a stream, sorted set or etc.

0 Answers
Related