I'm trying to save entities into redis. Each entity is a JSON (I don't do any operation on that json so I don't think (?) redis-json will help me here).

My service receives (over kafka-topic) alot of entities. some of them are updates of other entities.

Each entity has a "eventId" (it's like "category"),"bms" (it's like "entity-id"). Multiple entities with the same eventId may have different bms.

To be able to support consumer-groups (multiple kafka consumers) which all of them will save the entities in Redis: Each entity has a version (integer >=0).

Logic

Save the entity only if the new (from kafka) entity's version > current (inside redis) entity's version.

Data

My service consume upto 1k entities from the kafka per second. it means I need to SET 1k times at most per second.

Solutions

• I read in batch from the kafka-topic, so I have the ability to call redis with multi or pipeline.
• I assumed there is no way to avoid lua for atomicity when comparing "new" version with "current" version.

1. Lua Invokation Per Entity

local event_id = KEYS[1]
local bms = ARGV[1]
local new_version = tonumber(ARGV[2])
local new_offer = ARGV[3]

local bms_offer_hm_key = 'bms-offers--' .. event_id
local bms_version_hm_key = 'bms-version--' .. event_id

local current_version = tonumber(redis.call('HMGET',bms_version_hm_key, bms))

if current_version == nil or current_version < new_version then
  redis.call('HMSET',bms_version_hm_key, bms, new_version)
  redis.call('HMSET',bms_offer_hm_key, bms, new_offer)
  return 1
else
  return 0
end

Performance:

• redis 1k insers -> 0.050s
• redis 2k insers -> 0.100s
• redis 10k insers -> 0.400s

2. Lua Invokation for all Entities

local j=1
for i=1, #KEYS do
  local event_id = KEYS[i]
  local bms = ARGV[j]
  local new_version = tonumber(ARGV[j+1])
  local new_offer = ARGV[j+2]

  local bms_offer_hm_key = 'bms-offers--' .. event_id
  local bms_version_hm_key = 'bms-version--' .. event_id
  
  local current_version = tonumber(redis.call('HMGET',bms_version_hm_key, bms))
  
  if current_version == nil or current_version < new_version then
    redis.call('HMSET',bms_version_hm_key, bms, new_version)
    redis.call('HMSET',bms_offer_hm_key, bms, new_offer)
  end
  j=j+3
end

Performance:

• redis 1k insers -> 0.200s

3. Lua Invokation Per Entity - Single HM

• Now I'm using HSET instead of HMSET. Thanks to @raina77ow.

local molly_event_id = KEYS[1]
local bms = ARGV[1]
local new_version = tonumber(ARGV[2])
local new_offer = ARGV[3]

local key = 'molly-offers::' .. molly_event_id
local bms_sub_key = "bms::" .. bms
local version_sub_key = "version::" .. bms

local current_version = tonumber(redis.call('HMGET',key, version_sub_key))

if current_version == nil or current_version < new_version then
  redis.call('HSET',key, version_sub_key, new_offer, bms_sub_key, new_version)
  return 1
else
  return 0
end

Performance

• redis 1k insers -> 0.040s
• redis 2k insers -> 0.080s
• redis 10k insers -> 0.400s

More Performance & Info

My above metrics are based on docker-redis ontop of macbook pro:

macos catalina 10.15.1 (19B2093)
2.6 GHz 6-Core Intel Core i7
16 GB 2667 MHz DDR4

When trying to run 1k random sets (with the same entities) (without lua script):

• 1k insers -> 0.025s
• 2k insers -> 0.050s
• 10k insers -> 0.211s
• 45k insers -> 1.000s

I'm not in production yet, but when I will, redis will run ontop of k8s pod.

• single redis instace
• AOF + after every write
• I'm still investigating Redis Labs - Your Cloud Can’t Do That: 0.5M ops + ACID @<1ms Latency!

Questions

1. Is there a way to make the lua script faster? I'm open for any change!
2. I afriad blocking other clients from reading the entities while the lua scripts are blocking redis. multi-node will create additional distributed-logic complexity which I would like to avoid if possible.