We are experiencing intermittent DynamoDB execution times, taking consistently just over 10 seconds. This is happening roughly once every hour or three.
- The DynamoDB operations are called from a Node.js Lambda (either getItem or updateItem)
- The AWS SDK DynamoDB instance instantiation happens once outside of the Lambda handler
- The Lambda and DynamoDB are in the same VPC with a DynamoDB endpoint enabled
- The Lambda is triggered by a DynamoDB stream from another DynamoDB table which can happen multiple times a second
- The tables are set to PAY_PER_REQUEST
- The Partition Key is unique and we are getting concurrency of 4 Lambdas
- We have given the Lambda 2560MB of memory
- We have tried setting the BATCH_SIZE of the stream from 1 to 10 but it never prevents the intermittent long execution time
- We have enabled AWS SDK logging and can see on the long spike times that the DynamoDB call has been retried once - the duration on these occasions is consistently just over 10 seconds
- We have set the AWS SDK timeout from 5 secs to 2 secs and still see the 10 second delay
- We have maxRetries on the table set to 10
The cause for the consistent 10 second timing of the long spikes is hard to figure out. If anyone has any other idea to try out or advice it would be much appreciated. We are also liaising with DynamoDB, Lambda and Networking AWS support departments but so far they haven't figured out what is going on.
Much thanks,
Sam