Camel RabbitMQ connector reads thousands of message before using them

user_345070 0
Viewed 513

In my app, we are using a Camel route to read messages from a RabbitMQ queue. The configuration looks like that :

from("rabbitmq:myexchange?routingKey=mykey&queue=q")

The producer can send 50k messages within a few minutes, and each message can take 1 second or more to process.

What I can see is that that ALL messages are consumed very fast, but the processing of this messages can take many hours. Many hours of processing is expected but does that mean that the 50k messages are stored in memory ? If so, I would like to disable this behavior because I don't want to loose messages when the process goes down ... Actually, we are loosing most of the messages even when the process stays up, which is even worse. It looks like the connector is not designed to handle so many messages at once, but I cannot say if it is because of the connector himself or because we did not configure it properly.

I tried with the option autoAck :

from("rabbitmq:myexchange?routingKey=mykey&queue=q&autoAck=false")

This way the messages are rollbacked when something goes wrong but keeping 50k messages unacknowledge at the same time does not seem to be a good idea anyway...

apache-camel rabbitmq
1 Answers

There are a couple of things that i will like to share.

  1. AutoAck - Yes in case when you want to process the message ( after receiving it ) you should set AutoAck to False and explicitly acknowledge the message once it is processed.

  2. Setting Consumer PreFetch - You need to fine tune the PreFetch size , the pre fetch size is the max number of messages which RabbitMQ will present to the consumer in a go i.e. at the most your total un-acknowledged message count will be equal to the Pre Fetch size. Depending on your system if every message is critical you can set the pre fetch size to 1 , if you have multi threaded model of processing messages you can set the pre fetch size to match the number of threads where each thread processes one message and likewise.

In a way it acts like a buffer architecturally. If your process goes down while processing those message any message which was un acked before the process went down will still be there in the queue and the consumer will get it again for processing.

user_60515960
Related