I want to log as many platform produced actions as possible. (Kind of embracing event-sourced architecture but from the other way around). For example, I want to log when and how a report was created, or when and how a contact was deleted, or when an agent logged-in and logged-out.
Since our application tries to use event-based microservices decoupling (using AWS EventBridge) I have figured to just emit events for any actions I want to log. So, for example, an agent logged-in, an event is emitted, an agent created a report, an event is emitted (you get the idea).
Now I have created a catch-all-events rule that forwards any event to AWS Kinesis firehose. The firehose saves events in batches to the S3 bucket. Now to analyze those JSON events, I run an AWS Glue crawler on the bucket to produce schema. And lastly, I run AWS Athena queries to make sense of all that data I get.
This is the visualization of how events are being analyzed:
Now for you just to understand what kind of events/data can be found in that S3 bucket:
{"version":"0","id":"c1d9e9a4-25a2-a0d8-2fa4-b062efec98c4","detail-type":"OneTypeee","source":"OneSource","account":"123456789","time":"2021-01-17T12:35:17Z","region":"eu-central-1","resources":[],"detail":{"Key1":"Value1"}}
{"version":"0","id":"c13879a4-2h32-a0d8-9m33-b03jsh3cxxj4","detail-type":"OtherType","source":"SomeMagicSource","account":"123456789","time":"2021-01-17T12:36:17Z","region":"eu-central-1","resources":[],"detail":{"Key2":"Value2", "Key22":"Value22"}}
{"version":"0","id":"gi442233-3y44a0d8-9m33-937rjd74jdddj","detail-type":"MoreTypes","source":"SomeMagicSource2","account":"123456789","time":"2021-01-17T12:45:17Z","region":"eu-central-1","resources":[],"detail":{"MagicKey":"MagicValue", "Foo":"Bar"}}
And there are thousands and thousands of these lines in thousands and thousands of files. And there are hundreds of different types of events that contain different data. They are separated into different folders for the year, month, day, hour (this is done by the Kinesis Firehose).
Now the problem I run into, that every partition gets a slightly different schema and when I try to run Athena SQL queries I get this type of error (this is an example):
Your query has the following error(s):
HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced. The column 'detail' in table 'xxx-alleventscrawlerdatabase-dev.xxx_allevents_dev' is declared as type 'structyes:string,key:string,sum:int', but partition 'partition_0=Events2021/partition_1=01/partition_2=17/partition_3=12' declared column 'detail' as type 'structkey:string,sum:int'.
I have tried different crawler configurations but with no luck.
So my questions are:
- In general - is this a good approach to log and analyze your events data?
- How do I have a unified schema that covers all different keys and does not produce HIVE_PARTITION_SCHEMA_MISMATCH exception?