Route messages based on service instance unique id

userTvHX 0
Viewed 41

I have multiple instances of service A deployed where each service produces the request X events to SQS, which triggers the lambda functions and produces the various response events for that request submitted by particular service A instance. I would like to deliver these response events stored in a ActiveMQ topic such that each service A instance should only get the response events of a request it is submitted to queue. I can scale the number of service A instances horizontally.

For example, I have 2 instances of service A named instance1 & instance2. the instance1 submits the request to SQS with request id 1 and the instance2 submits the request to same SQS with request id 2 (request id is unique). The SQS will trigger the lambda functions, will produces the response events for request id 1 & 2 after lambda processing, stores them in a ActiveMQ topic (single one). The instance 1 & 2 are subscribers for this topic and the instance 1 should only get the response events of the request 1 and similary for instance 2.

Can we write a filtering logic for this? or can we fulfil this with Redis pub-sub? Kindly help.

spring-jms publish-subscribe activemq jms redis
1 Answers

This sounds like a request-reply pattern. There are several patterns for doing request and reply with ActiveMQ.

  1. Shared request destination, separate response destination(s)
  2. Shared request destination, shared response destination(s)
  3. Separate request destination, separate response destination(s)

For #1:

i. Producer sets up a consumer on the response queue (or topic) ii. Publish requests to a queue (or topic) iii. Set the name of the response queue (or topic) in the JMSReplyTo header on the message iv. Consumer receives the request, processes the messages and publishes response to queue (or topic) based on the value from the JMSReplyTo header

For #2:

i. Producer sets up a consumer on the response queue (or topic) using a selector to match the message based on a header value ii. Publish requests to a queue (or topic) iii. Consumer receives the request, processes the messages and publishes response to queue (or topic) and sets a value for the header field on the message

For #3:

.. basically the same as the others, but w/ separate request queues.

#1 is generally the most scalable, observable, and preferred option. You can also use 'disappearing' queues for request & response queues using ActiveMQ's destination garbage collection (esp useful for option #3).

Important-- you mentioned 'topic' and 'storing messages'. Keep in mind that topics do not store messages by default in ActiveMQ. You may want a queue instead.

Matt Pavlovich0
Related