Redis doesn't update when many requests at once

Viewed 1297

I have a Node.js application and I am trying to use Redis cache to keep count of the amount of requests made. This is just a proof of concept to see if Redis is the right tool for me, but I am a bit disappointed by the outcome and I'm wondering if Redis is just performing really badly or if there is a flaw in my code.

I have the following code:

const express = require('express')
const Redis = require('ioredis')

const client = new Redis(6379, 'redis');

client.on('connect', function() {
    console.log('Redis client connected');
});

client.on('error', function (err) {
    console.log('Something went wrong ' + err);
});

const getRedisCount = async (question_id) => {
    const count = await client.get('votes_' + question_id);

    if(count) {
        return count;
    }

    await setRedisCount(question_id, 0)
    return 0;
}

const setRedisCount = async (question_id, count) => {
    await client.set('votes_' + question_id, count)
}

const app = express();
app.use(express.json());

// post a vote
app.post('/vote', async (req, res) => {
    let count = await getRedisCount(req.body.question_id)
    count++
    await setRedisCount(req.body.question_id, count)

    res.status(200).send({ status: 200, data: count });

})

app.listen(5000, () => {
    console.log('Vote service listening on port 5000')
})

Now this all works as expected when I just do a single post request in Postman. the counter updates as expected.

But when I run a test in JMeter where I do 150 requests at once, I get a count of around 20-30 after the last request, instead of the expected 150.

So my question is - is this expected behavior, or is this a fault in my code ? and in both cases - how to fix it ?

3 Answers

This maybe something to do with your redis server config.

You can run MONITOR from the redis cli on your host machine.

Usually you connect via the cli for example:

https://redis.io/topics/rediscli

redis-cli -h <your server ip> -p <your server port> -a <if you have a server password enabled>

$ redis-cli -h host -p port -a password

Default port is 6379

This will show all of the interactions with the redis server, however don't run this as always running it has a performance hit but great for troubleshooting.

https://redis.io/commands/monitor

 $ redis> monitor
 1339518083.107412 [0 127.0.0.1:60866] "keys" "*"
 1339518087.877697 [0 127.0.0.1:60866] "dbsize"
 1339518090.420270 [0 127.0.0.1:60866] "set" "x" "6"
 1339518096.506257 [0 127.0.0.1:60866]

This should give you an indication of the issue.

You can also run the INFO command from the redis cli to see server stats and information such as memory usage.

https://redis.io/commands/info

redis> INFO
 # Server 
 redis_version 999.999.999
 redis_git_sha1:3c968ff0
 redis_git_dirty:0
 redis_build_id:51089de051945df4
 redis_mode:standalone 
 os:Linux 4.8.0-1-amd64 x86_64 
 arch_bits:64 
 multiplexing_api:epoll
 atomicvar_api:atomic-builtin 
 gcc_version:6.3.0
 process_id:8394
 </snip>

Also use the INCRBY command from your redis client in your code otherwise will need to increment your count value before setting it which is not consistent with the async pattern.

https://redis.io/commands/incrby

I think your problem is because this is asynchronous right?

So the counts are out of sync when they get posted since some may be awaited before others are.

Either way you mighty want to try and post them synchronous to see if that makes a difference then you know that is the issue and would be better off using the redis INCR commands for this purpose.

Sounds like an atomicity problem.

// post a vote
app.post('/vote', async (req, res) => {
    let count = await getRedisCount(req.body.question_id)
    count++
    await setRedisCount(req.body.question_id, count)

    res.status(200).send({ status: 200, data: count });

})

Let assume the count was 20, then many of your 150 requests will read 20 as their value.

    let count = await getRedisCount(req.body.question_id)

the next statement will increment and set count to 21 and the next line will try to save it to redis.

This is a standard race condition, better explained https://en.wikipedia.org/wiki/Race_condition#Example

You would have to implement a lock either in your code or use one provided by redis to resolve this.

The lock would essentially queue every request to increment one after another avoiding the overwriting of the values.

Not sure if this would work, but perhaps use a GETSET instead? I'm in the process of trying this out, as I have the same issue.

https://redis.io/commands/GETSET

Related