Symfony Messenger different consumers for different app servers

user_2737626 0
Viewed 377

I have a Symfony application that is running on several servers behind a load balancer. So I have separate hosts www1, www2, www3, etc.

At the moment I'm running messenger:consume only on www1, in fear of race conditions and potentially messages being handled twice.

Now I have a scenario where I need to execute a command on each host.

I was thinking of using separate transports for each host and running messenger:consume on each, consuming only messages from its respective queue. However I want the configuration to be dynamic, i.e. I don't want to do another code release with different transports configuration when a new host is added or removed.

Can you suggest a strategy to achieve this?

symfony-messenger symfony
1 Answers

If you want to use different queues and different consumers... just configure a different DSNs for each www, stored on environment variables (not code). Then you could have different queues or transports for each server.

The Transport Configuration can include the desired queue name within the DSN, and best practice is to store that configuration on an environment variable, not as code, so you wouldn't need "another code release with different transports config when a new host is added or removed". Simply add the appropriate environment variables when each instance is deployed, same as you do with the rest of the configuration.

framework:
    messenger:
        transports:
            my_transport:
                dsn: "%env(MESSENGER_TRANSPORT_DSN)%"

On each "www" host you would have a different value for MESSENGER_TRANSPORT_DSN, which would include a different queue name (or a completely different transport).

You would need to create a separate consumer for each instance, with a matching configuration (or run the consumer off the same instance).

But if all the hosts are actually running the same app, generally, you'd use a single consumer, and al the instances should publish to the same queue.

The consumer does not even need to run on the same server than any of the web instances, simply be configured to consume from the appropriate transport/queue.

user_14265390
Related