Dynamodb Autoscaling not working fast enough

user_3176550 0
Viewed 7139

I'm running a simple api that gets an item from a dynamodb table on each call, I have auto scaling set to a minimum of 25 and a maximum of 10 000.

However if I send 15 000 requests with a tool like wrk or hey, I get about 1000 502s,

  • dynamodb's metrics show that reads are throttled
  • the scaling activities log on the table shows that the RCUs were scaled to 99 but not more than that
  • lambda logs show that the function starts to take longer, it usually takes about 20ms to run, but the function starts to run for 500.1500,3000 ms and start timing out (I'm assuming that's caused by the throttling)

Why isn't the autoscaling working better? It only scales upto 99RCUs but my max is 10, 000.

amazon-dynamodb
2 Answers

We ran into the same problem when testing DynamoDB autoscaling for short amounts of time, and it turns out the problem is that the scaling events only happen after 5 minutes of elevated throughput (you can see this by inspecting the CloudWatch alarms the autoscaling sets up)

This excellent blog post helped us solve this by creating a Lambda that responds to the CloudWatch API events and improves the responsiveness of the alarms to one minute: https://hackernoon.com/the-problems-with-dynamodb-auto-scaling-and-how-it-might-be-improved-a92029c8c10b

user_4139030

from: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html

What you defined as "target utilization"?

Target utilization is the ratio of consumed capacity units to provisioned capacity units, expressed as a percentage. Application Auto Scaling uses its target tracking algorithm to ensure that the provisioned read capacity of ProductCatalog is adjusted as required so that utilization remains at or near 70 percent.

also, i think that the main reason that autoscale not works for you, is because your work might not stay elevated for a long time:

"DynamoDB auto scaling modifies provisioned throughput settings only when the actual workload stays elevated (or depressed) for a sustained period of several minutes"

DynamoDB auto scaling modifies provisioned throughput settings only when the actual workload stays elevated (or depressed) for a sustained period of several minutes. The Application Auto Scaling target tracking algorithm seeks to keep the target utilization at or near your chosen value over the long term. Sudden, short-duration spikes of activity are accommodated by the table's built-in burst capacity. For more information, see Use Burst Capacity Sparingly.

user_32184820
Related