DynamoDB removals and updates when streaming data to S3 in Parquet files

user_1317697 0
Viewed 427

We have created a quick proof-of-concept to stream the changes in a DynamoDB table to a Lambda function. This function does some basic ETL-processing and puts records on a Firehose. This Firehose writes Parquet files to S3, for some Athena querying. This works extremely well and easy for inserts, but I was wondering if there are tips or best practices for dealing with removals or updates.

I remember vaguely from a past conference that some people run batch jobs to compact the data and delete marked data, but I cannot find any details on this. Change Data Capture is another term, but I've yet to see clear examples of this. Is there a way of dealing with updates in this streaming setup?

UPDATE: I might have phrased my question poorly since a lot of the focus is going to S3. I'm really looking for a flow to process the incoming removals and updates in the DynamoDB stream, save them somewhere and then periodically update the Parquet file to reflect these updates and removals. Maybe some compaction will be needed as well, but that's "nice to have" at this stage in my experiment.

amazon-kinesis-firehose parquet amazon-web-services streaming
2 Answers

From what I understood by your question is that you want to deal with files on s3 and hence I would suggest you to use AWS Glue. you can create jobs as per your requirements.

If you use an Amazon S3 bucket as your data source, AWS Glue Studio detects the schema of the data at the specified location from one of the files, or by using the file you specify as a sample file.

Please refer to this: https://docs.aws.amazon.com/glue/latest/ug/edit-jobs-source-s3-files.html

Also refer to: https://aws.amazon.com/blogs/database/simplify-amazon-dynamodb-data-extraction-and-analysis-by-using-aws-glue-and-amazon-athena/

user_117793940

I think it's worth considering S3 versioning here too. If you do that, in your event message structure here, the s3.object.version key will exist. That way you'll be able to see if the entry was an edit or create, and for deletes you would look at: eventName. Otherwise you could have a system whereby you will only do a HTTP Post for creates, and PUTs for updates as that is also indicated in the eventName

If those options don't work for you, you would have to implement your own mechanism for detecting if the PUT was a new object or not, and this is system dependent how to do this. It may be best using S3 metadata, maybe it's easier in your own system, or using a table DynamoDB it really depends.

user_40332920
Related