RabbitMq Dynamically Add/Remove Queues/Consumers

user_10184876 0
Viewed 3975

I have an application that receives messages of three types from multiple users at same time and I am handling it using 3 queues/consumers and 3 exchanges. My problem is that when there are thousands of messages from one user in queues then other users are waiting.

I am looking for a solution to execute jobs of every user in parallel. I can create queues dynamically for every user but that's not a good solution because there will be hundreds of queues and consumers. How can I remove idle consumers/queues automatically. Can I use celery and redis for this problem in any way?

celery task-queue rabbitmq message-queue
2 Answers

It is not a good idea to use many queues in RabbitMQ. Queue is a single thread on RabbitMQ server.

Your problem is "thousands of msgs from one user will block other users", I think the solution is use Priority Queue.

Each user has a counter to record how many messages has been sent in a period time, if this value is large, the messages published by the user will have low priority. So if many messages are waiting in the queue, the messages from other user will be first sent to your consumer jobs. In this situation, you should set a appropriate "prefetch" on consumer's channel.

user_97978890

you have to add more queues in order to scale and to be consistent you can use a hashing distribution.

You can use plugins as https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange or you can do it manually, with more queues you can scale if one node is not enough you can create a cluster (https://www.rabbitmq.com/clustering.html) and distribute the load across more nodes.

One consumer for a queue, if you need the sequentiality.

if you don't need the sequentiality you can add more consumers to the same queue, so in this way, you can scale using more queues, more nodes, more consumers

user_18581160
Related