Pagination with AWS API Gateway + DynamoDB

Question

I have an API Gateway GET endpoint that scans a DynamoDB table and retrieve results according to a Limit parameter:

requestTemplates:
          application/json: >-
            {
              "TableName": "employee",
              "Limit": 2
            }

It is properly working and the response for this request when I send a limit = 2 is:

{
   "Count":2,
   "Items":[
      {
         "id":{
            "S":"18"
         },
         "department":{
            "S":"sales"
         },
         "name":{
            "S":"Roger"
         }
      },
      {
         "id":{
            "S":"16"
         },
         "department":{
            "S":"technology"
         },
         "name":{
            "S":"Petterson"
         }
      }
   ],
   "LastEvaluatedKey":{
      "id":{
         "S":"16"
      }
   },
   "ScannedCount":2
}

The problem is: I have 20 records stored in this table, and the ScannedCount and Count are both equal to 2.

I really need to know the count of the total amount of records I have stored in order to make a pagination frontend componenet to work.

I've looked through the documentation and a see that the expected result for this request would be ScannedCount = 2 and Count = 20.

Is there a way to have it? Thanks a lot.

user_290036 · Answer

If you have only the direct integration api-gw <-> dynamodb integration, check describe - https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DescribeTable.html and do two calls to get the desired data.

But I recommend using a Lambda in between Api Gateway and DynamoDB, which will facilitate the pagination.

user_1357094 · Answer

This probably isn't ideal, but if you can update your DynamoDB table schema to include an additional total attribute on each item (and implement corresponding count management of that attribute), then that would facilitate just one API Gateway request (still no Lambda or other type of repository layer compute necessary as in the Question).

I say probably not ideal as the complexity to manage [to some degree] accurate count of total is not necessarily trivial (>= 0). Depending on your context (for examples):

total might not be updated often, so maybe the complexity is less severe
Or else maybe total is updated frequently, so maybe that high updating chattiness makes complexity more severe

Related to those examples also is the idea of Data Freshness: Requiring "fresher" data would be when I referred there would be more chattiness complexity. That being said, even if the underlying data being counted itself updates frequently, updating total can be intentionally "stale" (as opposed to fresh) and therefore then reduce chattiness still; although the managing of anything other than [maybe relatively] optimal freshness is a facet of complexity in itself.

So in general, choosing to architect a solution involving this sort of top-level total should be confirmed worth the type of management complexity I've discussed here.

Pagination with AWS API Gateway + DynamoDB

2 Answers