What are good options for cloud storage of audit data? (written continuously, rarely queried)

Orion_Edwards_234 0
Viewed 514

I'm working on something which wants to store a lot of "audit record" style data. Such as "user created a thing", "user deleted a thing", "user had 27 things". We want to store it in the cloud (AWS or possibly Azure). Ideally I'd like to use a cloud-native service such that I don't have to worry about things like scaling up server instances, sharding, etc.

This kind of data is mostly a continuous stream of writes, and every so often a big "report" style query where we summarize or count a large block of data over a given date range. We need to query individual records approximately never.

The go-to for "cloud database in AWS" seems to be DynamoDB, however the pricing model, and the articles I've been reading seem to imply it is designed for more of a transactional model. Writes are an order of magnitude more expensive than reads, which is the opposite to my use case.

Those of you with deep cloud knowledge - please: What service(s) would you use for storing this kind of data and running reports on it? Is DynamoDB a bad choice here?

amazon-dynamodb cloud amazon-web-services architecture database
2 Answers

DynamoDB is probably a bad choice. Its big selling points are linear scaling and high levels of resilience. It comes with a pretty high price tag for transactions. If you're writing a lot of data it gets expensive quickly.

You mention 'written continuously' and 'rarely queried', which immediately makes me think S3 and Athena might be the best choice here.

S3 is the well known AWS 'blob store', its very cheap to keep data there. Athena is a serverless query technology that can utilise S3. Being serverless, you only pay for the queries you run, which sounds important in this case as you only query rarely.

user_49855800

In Azure you can stream your events to an Event Hub as they occur and then create a simple Logic app to move the events over to a Log Analytics workspace for safe keeping and any querying and reporting per your requirements.

There are Event Hub client libraries for several languages: https://docs.microsoft.com/en-us/azure/event-hubs/sdks

And several simple examples to follow for moving the data over to Log Analytics - a simple search will give you plenty of examples, so I won't link to any specifics here.

The max retention limit in Log Analytics is 730 days (2 years) (https://docs.microsoft.com/en-gb/archive/blogs/canberrapfe/change-oms-log-analytics-retention-period-in-the-azure-portal), so if you need to keep your logs around longer than this you'd need to explore putting it into a Data Lake instead of Log Analytics. This feels a bit 'heavy' to me compared to Log Analytics, but it really depends on your retention requirements.

user_140418390
Related