I just upgraded from hosting redis on an EC2 to using Elasticache with Cluster Mode Enabled. However, as I'm updating my code to use redis.RedisCluster I'm noticing some insane performance issues. I have some code that just gets all the keys for a certain pattern and deletes them. Previously I had no performance issues. But now, I see it takes over 3 minutes to do the scan, regardless of the output.
The code I'm using is here:
def get_keys(redis_conn: redis.RedisCluster, pattern: str):
key_list = []
start = time.time()
for key in redis_conn.scan_iter(pattern, target_nodes=redis.RedisCluster.ALL_NODES):
key_list.append(key)
end = time.time()
print(f'got {len(key_list)} keys in {end-start} seconds')
return key_list
And the output I get from 2 patterns are:
got 26246 keys in 194.3401861190796 seconds and got 5064 keys in 195.87923407554626 seconds.
Secondly, sometimes when I run this on docker, the process is just killed unexpectedly. I'm assuming this is some issue of reading the whole cluster into memory and python garbage collection but I'm not 100% sure.
How do people get all the keys across the cluster in Elasticache?