Imagine that I want to design an application that analyzes user events and provides some real time analytics dashboards to those users. Let's assume event ordering within the actions of each user is important.
Since I need to preserve the order of events for each user, and I also want to aggregate data based on each user, it makes sense to have a shard scheme that uses the UserID as partition key, so that all events of a single user are sent to the same shard, preserving ordering and making analytics faster since all data can be processed from the same shard.
Now, in this scenario, how would you handle hot users?
- I assume you can narrow down the hash range of the shard/shards that have some hot users, so those shards process less partition keys but with higher load of events each
- If you get to the point at which you are processing a single user and it is still too much for the shard, what would you do? Splitting across multiple shards would break ordering of events for that user, and would make analytics slower to compute. Also, not sure how could you even do that if you are using the UserID as partition key, you would probably need to add something else to it. Is there any solution to this scenario?
Thanks!