We have analytical ETL store model results on a Snowflake table (2 columns: user-id, score) We need to use that info in our low latency service, which snowflake is not suitable for that latency. I thought about storing that table on a Redis collection. I would like to have some idea of how to keep the Redis in sync with the table. any other solution for the latency is also welcomed
1 Answers
well it depends on how frequently you snowflake data is updated, what process is updating the data (snowplow or some external tool that you can hock into), what latency you want, are prepared between the snowflake data change, and redis having the values.
You could and a task to export the changes to a S3 and then have a lambda watching the bucket/folder, and push the changes into redis.
You could have your tool that loads the changes, pull the changes out and push those into redis. (we did a form of this)
You could have something poll the snowflake data (seems the worst idea) and push changes into redis. Well if you are polling the main table, this sounds bad, but you could also have a multi-table insert/merge command, thus when you are updating the main table, insert into a changes or stream, and thus read from this in you redis sync.
Related