Sorry for the long read. Technical info: Windows environment, MSMQ, .NET Framework
I have many mobile devices sending data to my server about events that happened on that device. Events are sent to my servers (multiple receiving servers, one DB) in the order they happened and on the server are sent to a centralized message queue. I have multiple servers each running a multi-threaded windows service that each thread pulls each time a message from the queue, processes it and saves it to the DB.
Usually the mobile devices are online and as events happen once every few minutes everything is saved in order to the DB as the servers have enough time to process the message before the next one from the same device arrives. but sometimes a device could be offline for a while and when it comes back online, it sends a few messages rapidly to the server (the simplest example is that the queue will now have messages #1 and #2 at the same time for this device - in the queue they are in order). Then two servers each pull a message from the queue and because of server load or thread speed message #2 could be saved to the DB before #1 is saved.
I tried solving this by implementing some kind of locking on the mobile device side so that it will acquire a “lock” from the server before sending message #1, then the lock is released after the server processes message #1 or the lock is released after a long timeout (if the message got lost somewhere) and then the device can acquire a lock to send message #2. It works but since the device is unaware of the processing time that it will take for #1 to be saved, it either has to poll the server a lot (which is ineffective and will drain the battery) or it tries to acquire a lock every 1-2 minutes which makes sending logged messages from the device really slow (the server could process it in 2 seconds so the device is just sitting idle for too long not sending the next message). More over, this requires the device to be online for longer as it sends messages to the server one by one.
I want to explore another option which is allowing the device to send all its data without acquiring a lock, thus releasing the mobile device online requirement as fast as possible. Incoming messages will be put into a message queue and then I need some solution on how to process them in a multi-server, multi-threaded environment, making sure messages are processed in the order they arrived per mobile device and as fast as possible. I thought about a single dispatcher thread reading from the incoming message queue and splitting the messages up into multiple queues - one for each server I have. The split will be based on the device that sent the message, so messages from the same device will be sent to the same queue (they still remain in order). Then on each server have one thread read from the server specific incoming queue and put the incoming messages in internal queues - one for each thread processing messages - messages from the same mobile device will be in the same internal queue. Then have the multiple threads read each from its own queue.
Any thoughts on a better design?
Thanks