How to store large arrays in DynamoDB

user_14550352 0
Viewed 144

I am new to DynamoDB and I am curious what is the best way to store potentially large arrays.

I have a user object that looks like this:

UserId: String
Watching: Card[]
Listings: Card[]

I am aware there is a limit to the size of objects in Dynamo - I think 1MB? Therefore I think if a user had many listings it might go past this limit. What would be the best practice to store potential large arrays like this? Would it be to maybe store an array of CardIds and then do a second query to get cards from this?

dynamodb-queries amazon-dynamodb amazon-web-services arrays database
3 Answers

The limit of an object in DynamoDB is 400 KB, see DynamoDB Quotas.

For larger attribute values AWS suggests compressing of the attribute in formats such as GZIP, and store it in binary in DynamoDB. Other option would be to store the item in JSON format in S3 and store the key of this file in DynamoDB.

See: Best Practices for Storing Large Items and Attributes

Probably, a third option would be to split your array somehow, and create multiple entries in DynamoDB. Or try to create separate tables for separate attributes, obviously this wont solve the issue if for example Listings array's size is larger than the object limit itself.

user_76611190

One option is to use the Single Table Design approach where all users, watchings, and listings are in the same table.

User items would have a primary key of user#uid and sk of user#uid (the same as pk), each watching item would have a pk of user#uid and sk of watching#wid, and each listing would have a pk of user#uid and sk of listing#lid.

For example:

pk sk other attributes
user#1 user#1 yes
user#2 user#2 yes
user#42 user#42 yes
user#42 watching#12 optional
user#42 watching#29 optional
user#42 listing#901 optional
user#42 listing#472 optional

This approach has no real limit on the number of watching or listing relationships.

You can then query all items for a given user simply by issuing a query for pk=user#42 and that will yield the user and all associated watching and listing items (pagination notwithstanding). You can query all listings for a given user with pk=user#42 and sk begins_with("listing#").

Note that this will increase the table size because of the additional "user", "watching", and "listing" prefixes on attribute values so you may want to consider abbreviating those.

To quote Alex DeBrie:

The main reason for using a single table in DynamoDB is to retrieve multiple, heterogenous item types using a single request.

user_2714150

The table in dynamoDB, every row in the table is object.

The item in a row view it in json object format.

The row cannot support json array.

Split into 2 table. Store listing in table A and another in table B. Design the primary key and sort key correctly so that it can identified the records belong to particular user.

user_193488200
Related