How to do basic aggregation with DynamoDB?

Question

How is aggregation achieved with dynamodb? Mongodb and couchbase have map reduce support.

Lets say we are building a tech blog where users can post articles. And say articles can be tagged.

user
{
    id : 1235,
    name : "John",
    ...
}

article
{
    id : 789,
    title: "dynamodb use cases",
    author : 12345 //userid
    tags : ["dynamodb","aws","nosql","document database"]
}

In the user interface we want to show for the current user tags and the respective count.

How to achieve the following aggregation?

{
    userid : 12,
    tag_stats:{
        "dynamodb" : 3,
        "nosql" : 8
    }
}

We will provide this data through a rest api and it will be frequently called. Like this information is shown in the app main page.

I can think of extracting all documents and doing aggregation at the application level. But I feel my read capacity units will be exhausted
Can use tools like EMR, redshift, bigquery, aws lambda. But I think these are for datawarehousing purpose.

I would like to know other and better ways of achieving the same. How are people achieving dynamic simple queries like these having chosen dynamodb as primary data store considering cost and response time.

user_9671280 · Answer

Dynamodb is pure key/value storage and does not support aggregation out of the box.

If you really want to do aggregation using DynamoDB here some hints.

For you particular case lets have table named articles. To do aggregation we need an extra table user-stats holding userId and tag_starts.

Enabled DynamoDB streams on table articles
Create a new lambda function user-stats-aggregate which is subscribed to articles DynamoDB stream and received OLD_NEW_IMAGES on every create/update/delete operation over articles table.
Lambda will perform following logic

If there is no old image, get current tags and increase by 1 every occurrence in the db for this user. (Keep in mind there could be the case there is no initial record in user-stats this user)
If there is old image see if tag was added or removed and apply change +1 or -1 depending on the case for each affected tag for received user.

Stand an API service retrieving these user stats.

Usually aggregation in DynamoDB could be done using DynamoDB streams , lambdas for doing aggregation and extra tables keeping aggregated results with different granularity.(minutes, hours, days, years ...)

This brings near realtime aggregation without need to do it on the fly per every request, you query on aggregated data.

How to do basic aggregation with DynamoDB?

2 Answers