RabbitMQ cleanup after consumers

user_6035109 0
Viewed 157

I having problem handling the following scenario:

I have one publisher which wants to upload a lot of binary information (Like images), so instead I want it to save the image and upload a path or some reference to that file.

I have multiple different consumers which are reading from this MQ and do different things. In order to do that, I simply send the information in fan-out to some exchange and define several queues for each different consumers.

This could work just fine, except for the trashing of the FS. Since no one is responsible for deleting the saved images. I need some way of defining a hook to the time every consumer is done consuming a message from an exchnage? Maybe setting some callback for the cleanup of the message in the exchnage?

Few notes:

  1. Everything happens locally, we can assume that everything is on the same FS for simplicity.

  2. I know that I can simply let the publisher save the image and give FS links for the different consumers, but this solution is problematic, since I want the publisher to be oblivious to the consumers. I don't want to update the publisher's code every time a new consumer may be used (or one can be removed).

  3. I am working with python. (pika module)

  4. I am new to Message Queues, so if you have a better suggestion to get things done, I would love to learn about it.

rabbitmq python
1 Answers

Once the image is processed by consumer publish message FileProcessed with the information related to the file. That message can be picked up by another consumer which is in charge of cleaning up the messages, so that consumer will remove the file.

Additionally, make sure that your messages is re-queued in case of failure, so they will be picked up later and their processing will be retried. Make sure the retry count is limited, when the limit is reached route the message to dead letter exchange.

Some useful links below:

user_20679760
Related