Design DynamoDB for Large data using array. Over 400k

Viewed 598

I'm worked with node.js and for record event-time and event-data, I want to use AWS DynamoDB Table.

single data and one-key-data schema sample like under.

var want_to_update_data = [ // contain json more or same than 1
  { "event-time" : "2021-01-02 10:11:12", "event-data" : 19 },
  ...
];

var saved_data = { // DynamoDB Saved Data Sample with Key "ABCD"
  "Key" : "ABCD",
  "events" : [
    ...
    { "event-time" : "2020-12-28 09:30:17", "event-data" : 35 },
    { "event-time" : "2021-01-01 19:11:12", "event-data" : 16 },
    { "event-time" : "2021-01-02 10:11:12", "event-data" : 19 },
    ...
  ]
}

Upper data is only for sample. json contains more key-values.


For update single json, my server code likes under.

app.get('/insert', req, res) {
  var update_key = req.body.update_key; // "ABCD"
  var want_to_update_data = req.body.update_data; // [ {} ]

  var updateExpression = 'SET events = list_append(events, :dl)'; // dl means data-list
  var expressionAttributeValues = {':dl' : want_to_update_data }; // dl means data-list

  AWS.config.update({
    region: "region",
    endpoint: "endpoint",
    accessKeyId: "access_key",
    secretAccessKey: "secret_key",
  });

  var params = {
    TableName: "TableName",
    Key: update_key,
    UpdateExpression: updateExpression,
    ExpressionAttributeValues: expressionAttributeValues,
    ReturnValues: "UPDATED_NEW"
  });

  var dynamodbClient = new AWS.DynamoDB.DocumentClient();
  dynamodbClient.update(params, function(err, data) {
    if(err) { /* handle error */ }
    else { /* handle after update complete */ }
  });
});

Simply, It works, but AWS DynamoDB can't change over 4kb at once.
So, if array data is bigger than 4k, list_append not worked with ValidationException: Item size to update has exceeded the maximum allowed size error message.


I want to remain all events and want to sort event-time keys.

How can I change my DynamoDB schema?

Under code is just my guess.
Is it possible?

{
  "Key" : "ABCD",
  /*
  "events" : [
    ...
    { "event-time" : "2020-12-28 09:30:17", "event-data" : 35 },
    { "event-time" : "2021-01-01 19:11:12", "event-data" : 16 },
    { "event-time" : "2021-01-02 10:11:12", "event-data" : 19 },
    ...
  ]
  */
  "events" : { // is it possible events['some-date'] could be sort-key?
    ...
    "2020-12-28 09:30:17" : 35,
    "2021-01-01 19:11:12" : 16,
    "2021-01-02 10:11:12" : 19,
    ...
  }

}

/*
  var updateExpression = 'SET events = list_append(events, :dl)'; // dl means data-list
  var expressionAttributeValues = {':dl' : want_to_update_data }; // dl means data-list
*/

  // under 2 line is just for sample. json key is not fixed
  var event_time = '2021-01-02 10:11:12';
  var single_json = want_to_update_data[event_time];

  // if number-start key is not allow(event_time), prefix append could be like "D_2021~~"
  var updateExpression = `SET events.${event_time} = :dl`;
  var expressionAttributeValues = {':dl' : single_json }; // single json

Thank you for read my question.

I want to create new table for save my events json datas. (not S3. using DynamoDB)

2 Answers

DynamoDB has an item size limit of 400kb, including the attribute names.

According to the docs

Number of Values in List, Map, or Set There is no limit on the number of values in a List, a Map, or a Set, as long as the item containing the values fits within the 400 KB item size limit.

It sounds like your list of 4k events, plus whatever else you're saving in the item, exceeds this 400kb limit. You can check out this nifty calculator to get an idea of the size of your item.

Instead of storing events in a list item, you might want to store them in an item collection.

For example, instead of this:

enter image description here

You could store event data like this

enter image description here

If you need to fetch events by time, you might consider making the event_time your sort key.

enter image description here

You cannot store infinitely large amounts of data in a DynamoDB on single Key.

Just like what you wrote, periodically back up to S3.

In my case, I use redis-cache for check each array size and with node-schedule, back-up large datas to S3.

Related