DynamoDB | BatchGet | Get results in same order as provided Keys

user_1581803 0
Viewed 4277

I'm using AWS DynamoDB to store users.

Consider the following code:

let params = {
  RequestItems: {
    'users': {
       Keys: [
        {id: '1111'},
        {id: '2222'},
        {id: '3333'},
        {id: '4444'},
        ]
        }
      }
    };

Using the above params in a BatchGet will return the reqeusted users but in a random order!

Question: Is it possible to BatchGet users without losing the order defined in Keys?

documentclient amazon-dynamodb amazon-web-services
2 Answers

You'd have to sort the items once they are retrieved. As documented here, When designing your application, keep in mind that DynamoDB does not return items in any particular order.

I had the same issue in the recent past & had to write some extra code to sort the items the way I wanted.

Update 22-Aug-2019: I wanted to mention that you could switch to using query on a GSI with a sort key and retrieve sorted data if that is feasible in your case.

Use ScanIndexForward: true || false to sort the data in ascending or descending as needed.

More details here

user_29431740

As Gary Vernon Grubb correctly answered, the fact that BatchGetItem doesn't return items in order is documented. I just wanted to add that this "random order" doesn't happen just to annoy you, but is deliberate to lower latency:

If your batch asks to retrieve 100 items, DynamoDB will immediately start to retrieve all of them, in parallel. Each one may come from a different node in Amazon's cluster. Whatever response comes first, Amazon can send it back to you immediately. But this means you'll get the responses back in seemingly random order.

Had Dynamo insisted on returning the first-requested item first, if you're unlucky, this item may be the last one to become available - maybe the node storing it was unusually busy or unlucky - and only then could DynamoDB start sending your the entire response. This adds the latency of the slowest request to the latency of sending the entire batch, instead of overlapping the two latencies. Moreover, buffering and sorting the retrieved items would have been inefficient for the DynamoDB implementation, because the overall response can fairly long (https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html explains it can be up to 16MB), and buffering it in memory until it's all ready would be expensive. It's more efficient to just send every item when it becomes available.

Finally, please note that not only do you get responses in random order, it is also possible that you won't get responses for all your requests. If some of the requests were not processed, you'll get a list of them in UnprocessedKeys, and need to make those requests again. These don't have to be the last of your requests, they can be and subset of the requested keys. This case can happen because the total size of the response exceeds 16MB so some of it was not returned, or because you exceeded your provisioned capacity.

user_88912240
Related