How do I achieve one by one message delivery for each user individually (in parallel)?

user_5759211 0
Viewed 69

I need a queue/message broker that would allow me to receive by 1 message for each user.

Previously I've been using FIFO SQS and each user has its own message group id, which allows me to have 1 message inflight for each user. diagram

But I've ended up with a problem when I have more than 20.000 messages in the queue, because A FIFO queue looks through the first 20k messages to determine available message groups, which makes all messages above the 20.000 limit to be unavailable for processing.

Do you guys have any advice for a message broker/queue that would allow me to achieve the same behavior without having a limit of 20.000 messages?

I was thinking about using Redis Lists (where each user has its own list and I do an RPOP for each existent list)

amazon-sqs redis activemq rabbitmq message-queue
1 Answers

A couple options to overcome the consumer window problem you describe in ActiveMQ:

  1. Use Virtual Topics and have consumers register a selector. The broker will filter messages to each consumer into their own queue. This does not require any broker changes and is completely dynamic via the client setting the selector.

  2. Use Composite Destinations to filter messages to separate queues. This is static broker routing and requires server-side config change for any new routing rules. The clients do not need to be changed

Both approaches documented here: ref: https://activemq.apache.org/virtual-destinations

user_60845890
Related