Background
We have a need for messages to be processed in the order they were published. This is necessary to make sure all updates related to a particular entity are applied in order. Example: User updated their address 2 times in quick succession (could be on the UI OR a burst of messages from a batch job), we want to be able to apply Change 1 first and then Change 2 on top to make sure the data is correct. We achieved this by implementing a Hash Exchange and 20 "Single-Active Consumer" queues bound to this exchange. We listed all these queues in the micro-service config file and deployed it on a Kubernetes pod. This setup works well. We see a single consumer on each of the 20 sharded queues and messages are processed in the order they were published (for a given entity). Note that our microservice is NOT just processing messages from RabbitMQ. It is also serving REST requests from the users VIA the UI/Mobile.
Problem Statement
We are working on autoscaling our microservice when the current instance is getting heavily loaded. We want to spin up new instances to help share the load. Today when we do this, the new Kubernetes pod connects to the 20 sharded queues as expected but the consumers are in a wait state (Like they should be because our queue is defined as a Single-Active Consumer queue). Consequently, the messaging load stays on the original pod and the new pods are only helping with the web traffic but they cannot share the messaging load. What we want is a way to dynamically rebalance consumers between pods so that load is evenly spread across pods. Upon initial review, this might sound like a VERY narrow use case but if you dig a bit deeper it becomes clearer that this is a common use case.
- Being able to process messages in order is a very common requirement many people would have.
- Microservices on Kubernetes and autoscaling Up and Down based on load is also a very common scenario.
- Combining those two would naturally lead to having the ability to distribute messaging load across Pods is also something folks would be very interested in.
Given the current feature set offered by RabbitMQ and Spring, I can't see a way to implement this without writing a lot of code and in some cases ending up with a suboptimal solution. I would like to know if any one ran into this problem and how did you solve it. Any ideas/inputs in this matter would be greatly appreciated.