How to serialize `LastEvaluatedKey` from DynamoDB's Golang SDK?

Question

When working with DynamoDB in Golang, if a call to query has more results, it will set LastEvaluatedKey on the QueryOutput, which you can then pass in to your next call to query as ExclusiveStartKey to pick up where you left off.

This works great when the values stay in Golang. However, I am writing a paginated API endpoint, so I would like to serialize this key so I can hand it back to the client as a pagination token. Something like this, where something is the magic package that does what I want:

type GetDomainObjectsResponse struct {
  Items     []MyDomainObject `json:"items"`
  NextToken string           `json:"next_token"`
}
  
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
  // ... parse query params, set up dynamoIn ...

  dynamoIn.ExclusiveStartKey = something.Decode(params.NextToken)

  dynamoOut, _ := db.Query(dynamoIn)

  response := GetDomainObjectsResponse{}
  dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)

  response.NextToken := something.Encode(dynamoOut.LastEvaluatedKey)
  
  // ... marshal and write the response ...
}

(please forgive any typos in the above, it's a toy version of the code I whipped up quickly to isolate the issue)

Because I'll need to support several endpoints with different search patterns, I would love a way to generate pagination tokens that doesn't depend on the specific search key.

The trouble is, I haven't found a clean and generic way to serialize the LastEvaluatedKey. You can marshal it directly to JSON (and then e.g. base64 encode it to get a token), but doing so is not reversible. LastEvaluatedKey is a map[string]types.AttributeValue, and types.AttributeValue is an interface, so while the json encoder can read it, it can't write it.

For example, the following code panics with panic: json: cannot unmarshal object into Go value of type types.AttributeValue.

lastEvaluatedKey := map[string]types.AttributeValue{
    "year":  &types.AttributeValueMemberN{Value: "1993"},
    "title": &types.AttributeValueMemberS{Value: "Benny & Joon"},
}

bytes, err := json.Marshal(lastEvaluatedKey)
if err != nil {
  panic(err)
}

decoded := map[string]types.AttributeValue{}
err = json.Unmarshal(bytes, &decoded)
if err != nil {
  panic(err)
}

What I would love would be a way to use the DynamoDB-flavored JSON directly, like what you get when you run aws dynamodb query on the CLI. Unfortunately the golang SDK doesn't support this.

I suppose I could write my own serializer / deserializer for the AttributeValue types, but that's more effort than this project deserves.

Has anyone found a generic way to do this?

user_1513338 · Answer

OK, I figured something out.

type GetDomainObjectsResponse struct {
  Items     []MyDomainObject `json:"items"`
  NextToken string           `json:"next_token"`
}
  
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
  // ... parse query params, set up dynamoIn ...

  eskMap := map[string]string{}
  json.Unmarshal(params.NextToken, &eskMap)
  esk, _ = dynamodbattribute.MarshalMap(eskMap)
  dynamoIn.ExclusiveStartKey = esk

  dynamoOut, _ := db.Query(dynamoIn)

  response := GetDomainObjectsResponse{}
  dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)

  lek := map[string]string{}
  dynamodbattribute.UnmarshalMap(dynamoOut.LastEvaluatedKey, &lek)
  response.NextToken := json.Marshal(lek)
  
  // ... marshal and write the response ...
}

(again this is my real solution hastily transferred back to the toy problem, so please forgive any typos)

As @buraksurdar pointed out, attributevalue.Unmarshal takes an inteface{}. Turns out in addition to a concrete type, you can pass in a map[string]string, and it just works.

I believe this will NOT work if the AttributeValue is not flat, so this isn't a general solution [citation needed]. But my understanding is the LastEvaluatedKey returned from a call to Query will always be flat, so it works for this usecase.

How to serialize `LastEvaluatedKey` from DynamoDB's Golang SDK?

1 Answers