I have a channel exclusive for consumption and 50 queues, each with exactly one consumer. The channel has a prefetch value of let's say 5 which applies to each consumer of the channel.
My application needs to process messages exactly in the order it received them within each queue. Is it possible for a consumer of a queue to start processing the next message without ack'ing the earlier message because of the high prefetch value? I'm using NodeJS with ampqlib
The answer is that it depends on the language you're working in.
In JS, it will not be ordered if we do something async in the callback. To ensure it's ordered we can leverage a trick exploiting the fact that JS is single threaded. Here's the rough idea which you can extend to multiple queues using a hashmap.
beingProcessed = false;
.on("messageFromQueue", (msg) => { // sync callback
// very important that the callback here does no async work prior to inserting into the
// queue to guarantee that the next message won't be pushed first
queue.push(msg); // in memory queue
processSerially(msg);
})
async processSerially() {
if(beingProcessed || queue.size() === 0) {
return;
}
beingProcessed = true;
while(queue.size() > 0) {
await doSomeTaskAsyncWithMessage(queue.pop())
}
beingProcessed = false;
}