Is DynamoDB suitable for high transaction application?

user_718774 0
Viewed 504

I am writing horse racing odds aggregation application which will get data from difference bookies website. For a start I will get data from 3 websites(it can go up to more than 10 later) every 10 seconds. So in 3 websites case, there will be about 10,000 records(runners) each day and each record could be read 3 times every 10 seconds and updated if there are changes in odds.

  1. Is DynamoDB suitable that kinds of application or should I stick to RDBMS?
  2. Will there be any consistency issue with DyanmoDB when I am updating the odds(from different websites) for the same runner(record) simultaneously?
  3. The application may grow into other sports and racings and get data from more websites, will that incurred a huge cost with DyamoDB as it will do more read and write?

UPDATE - 21/07/2020 9:30AM Record structure will be something like below. There will be a few scheduled services running with each service taking care for a bookie. There are chances that a record will be read by services and updated simultaneously. Calculated column value will be based on the value of Bookies column. Hence, I want to be able to read the most recent value of Bookies column consistently.

  RUNNER       EVENTID  BOOKIE1     BOOKIE2     BOOKIE3     BOOKIE...    CALCULATED
  Runner 1     12345    Odds1       Odds2       Odds3       Odds...      Value
  Runner 2     67890    Odds1       Odds2       Odds3       Odds...      Value

UPDATE - 21/07/2020 12:20PM

After updating my post i get some numbers pop up in my head and DynamoDB seems to be very expensive. Here are my numbers, please let me know if anything is incorrect.

Assumptions:

  1. 10,000 Runner
  2. Every 10 seconds for a month roughly round up to 270,000 calls
  3. 3 bookies
  4. Assuming each record/item is under 4KB.
  5. One RCU can read 5.2million read per month (found somewhere)
  6. One WCU can read 2.5million read per month

RCU Required per month: (3 * 1,0000 * 270,000)/5.2 mill = 1558 RCU

WCU Required per month: (3 * 1,0000 * 270,000)/2.5 mill = 3240 WCU

cost-management amazon-dynamodb consistency amazon-web-services rdbms
3 Answers

DynamoDB is built for performance and scalability (specifically targeting reads), it has support for transactions.

In fact whilst a relational database uses the ACID model, DynamoDB as a NoSQL Key-Value uses the BASE model. This trades features like consistency (which guarantees a transaction has written to the disk before responding success) for the ability to have cut-edge performance.

You could definitely use DynamoDB but you would need to be aware of the limitations, for example you should not be trying multiple updates on the same item at the same time. You mention that you're doing this once every 10 seconds so could a process aggregate the changes then apply.

If you care for real time data then you will to use a strong consistent read to ensure that you're reading the most accurate data.

You can reduce some of your read consistency cost through DAX the built in caching layer that sits in front of DynamoDB.

Additionally if you have low usage period times DynamoDB provides auto scaling built into it, which can reduce the capacity you pay for (reads and writes) when you're quieter.

Other than this if you want performance whilst maintaining transactional writes, Redis an in-memory datastore supports transactions. There is an AWS managed version with ElastiCache.

There is of course the Relational DB option too, whilst this will allow transactional writes you will need to consider read performance (whether that's through a cache or through read only functionality).

Ultimately the choice comes down to you, each of these options have limitations but it comes down to how you expect to use it. DynamoDB will likely be the cheapest option but you need to consider the architecture for your expected demand.

user_134609330

DynamoDB is absolutely able to keep up with this kind of load and way more. In that regard, do not worry about DynamoDB

For consistency, DynamoDB does have the ability to do strongly consistent reads. In addition, so you know how writes work, DynamoDB acknowledges writes only once it gets to at least two of the three storage nodes for that partition. One of those two must be the leader node for that partition. Strongly consistent reads always come from the leader node.

As for cost, it depends on many factors and without knowing more about your workload and how it will grow, i am hesitant to guess. If you want to watch it, you can do a pay as you go with on-demand capacity mode on the table(s) and you only pay for exactly what you use.

user_34161350

The recommendation for high scalability for this kind of situation using DynamoDb is DAX ( https://aws.amazon.com/en/dynamodb/dax/ ), that makes Dyanamo suitable.

About the consistency, it will depend on your data model, and with it goes, but Dynamo with DAX handle this good, here is a link of consistency recommendations for DAX + Dynamo: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DAX.consistency.html

And finally, yeah, all the services that you use in any Cloud provider have an I/O quota or rate, so when the application grows the price will do it to.

user_114821760
Related