Let's take a multi-tenant microservices system monitoring some kind of user devices. Let's say I have 1000 clients with 2000 devices each. Each device generates events maybe a couple times per minute. Events from a particular device must be processed sequentially, but we can process events from different devices simultaneously.
The event processor should be scalable horizontally to support more clients and devices. Event processing will be cheap, so I'm expecting a single service instance to be able to process lots of events per second. Let's say 10K/s? So with 200 service instances we should be able to process 2M events/s.
I thought of using Azure ServiceBus to queue the events, and set the SessionId to, say, each device's id, in order to guarantee sequential processing per device. But this means I have potentially millions of sessions to process.
Where I need some guidance is around the number of concurrent sessions per service instance. ServiceBusSessionProcessorOptions.MaxConcurrentSessions defaults to 8. In order to do 10K events/s, if the sessionId is per device, we'd need on the order of 10K concurrent sessions actually. Is this a number that makes sense? Do we run into physical limitations like sockets etc.?
If I use something less granular, say 1 SessionId per client, then I can use a much lower number of concurrent sessions, but the load might not be spread as evenly. A given instance might end up processing messages from a couple of very busy clients.
What's a good granularity for SessionIds? Is it meant to be used at the individual device level, resulting in a very large number of concurrent sessions?