Writing only whitelisted items to DynamoDB

user_3139545 0
Viewed 91

I have a set (~1.000.000) of devices that can send messages, each device has an ID and can send multiple messages. A subset (~10.000) of the devices should be able to use my service and there IDs are put in a whitelist (DynamoDB table). Given a constant stream of messages I would like only the once in the whitelist to be persisted in a DynamoDB table.

The things I have been looking at are as follows:

  1. Naive way is to do the filtering in code when the lambda writes data to DynamoDB, not sure how well this will scale given that I first need to query the whitelist table to check which messages I then can write.
  2. What I would like to do use use DynamoDB conditional writes (or if there is some other smart functionality) to specify "write this item to the table only if it exist in the whitelist" and let the DynamoDB engine take care of the filtering.

My question is if there is some smart way to do this in DynamoDB?

amazon-dynamodb nosql amazon-web-services
2 Answers

This is both a blessing and curse of DynamoDB - at first, it's a pain, but after a short while DDB's simplicity starts to shine - you can learn its entire API in a short time.

Every problem is different and you've asked a few things in your question, so I'll just offer some points that will hopefully help:

  • You cannot query/reference another table in DDB, so there's no "smart" way to do it.
  • DDB is very fast, so I would not be worried about scaling on the DB's end.
  • Querying before writing is a perfectly acceptable model, just make sure you have an efficient index and that you're using GetItem or Query - in other words, do not use Scan.
  • If you really want to minimise calls, then you could consider creating a simple in-memory cache of whitelisted ID's. This may cause scaling complexity, depending on your infrastructure, but this can be easily addressed. If this is something you are considering, then I'd suggest looking at DDB Streams as a way to ensure caches are up to date. But to be honest, unless you actually have a throughput problem, this option would be overkill.
user_46326270

In general, there is no way to do what you want with a conditional update. A conditional update can only refer to the content of the same item being modified - not to other items or other tables. This is not a missing feature - this is really something that cannot be done for "free": While DynamoDB writes to an item it can inspect this item's previous value "for free", as part of the same operation. But there is no "free" way read a different item. It basically amounts to another read operation. So if you want to read this other item, you can do it yourself with a read operation - DynamoDB cannot really do this read any more efficiently than what an extra read request would do.

In some cases, if you only keep one item per id, there is an easy solution to your problem - just add another attribute to the item saying whether it's whitelisted, and then use this new attribute in a conditional update. But you can't do this if there are many items per id (e.g., the id is a partition key and you have an additional sort key). CQL (Cassandra and Scylla) have the notion of a "static row" (an item which belongs to the entire partition, not to a sort key within the partition) but DynamoDB doesn't have this feature. And even if it did, the extra read of the static row might have been cheaper than another read, but it would not be free.

user_88912240
Related