Laravel + Rabbit = Crontab vs Supervisor

Question

I've a php application (laravel 5.8) that implements a rabbitmq consumer (bschmitt/laravel-amqp).
When the application starts, to keep under control the consumer process, a supervisor process starts as well .
My supervisor conf file:

[program:message-consume]
process_name=%(program_name)s_%(process_num)02d
command=php artisan message:consume 
directory=/var/www
autostart=true
autorestart=true
numprocs=1
redirect_stderr=true
stdout_logfile=/var/www/storage/logs/supervisor.log
stopwaitsecs=3600
startretries=20

The consume command

public function consume($routingKey)
{
  Log::info("[MessagingService][consume] start consuming key: " . $ROUTING_KEY . ".*");
  Amqp::consume($QUEUE_NAME, function ($message, $resolver) {
    # my code...
    $resolver->acknowledge($message);
  }, [
      'routing' => $ROUTING_KEY . '.*',
      'queue' => $QUEUE_NAME,
      'exchange' => $EXCHANGE,
      'exchange_type' => 'topic',
      'exchange_durable' => false,
      'queue_force_declare' => true,
      'queue_exclusive' => false,
      'persistent' => true, // set true if consume forever
  ]);
}

Everything seems to be alright but I realized that, if there are no messages for a while, the application stop receiving messages!

The processes is still running and supervisor doesn't do anything.
What's going on? What I can see from rabbit dashboard is that there are no consumers connected to the channel.

The idea was to create a cron script that every minutes call rabbitmq api to check if there are consumers in the queue and, in case, restart supervisor.

I'm working with docker so I changed my startup.sh in order to add the cron

#!/bin/bash

supervisord

crontab /etc/cron.d/check-consumer-cron
cron -f

php-fpm

But I start asking me questions. Am I doing well? Am I missing something? Do I need supervisor?
If I change the startup.sh in the following way, what changes?

#!/bin/bash

php artisan message:consume 

crontab /etc/cron.d/check-consumer-cron
cron -f

php-fpm

user_157957 · Answer

By restarting Supervisor, you are tackling the problem from the wrong end. Supervisor's job is to keep your script running, you need to understand why that isn't happening.

I believe your problem actually has two parts:

Your consumer is disconnecting if idle for too long. This is likely a network timeout; configuring the "heartbeat" setting on the AMQP connection may be enough to keep it alive.
Supervisor isn't detecting this state and restarting your consumer. That's because it doesn't monitor the network state of your consumer, only whether the PHP script is still running. You need some code in your consumer that exits PHP rather than continuing to wait for a message it will never receive.

I don't know the Laravel wrapper, but generally a consumer works by having an infinite loop somewhere that looks a bit like this:

while ( true ) {
   $connection->waitForAndProcessNextMessage();
}

Each time a message comes in, or a timeout expires, the "waitForAndProcessNextMessage" method (whatever it's actually called) will return, and at that point you can decide whether to carry on round the loop, or stop and exit the script. You could put a status check there, or just see how long the script has been running:

while ( true ) {
   $connection->waitForAndProcessNextMessage();
   if ( ! $connection->isStillConnected() ) {
      break;
   }
   if ( minutesSinceConsumerStarted() > 30 ) {
      break;
   }
}

user_1622283 · Answer

I want to answers to my questions

Am I doing well? Am I missing something?
The package I was using is a wrapper on top of php-amqplib and this wrapper offers a limited documentation and, badly, no control over the inner structure. Of course I wasn't able to debug and check what was going on. Using directly php-amqplib and following the examples they provide I never faced again that behaviour.

Do I need supervisor? If I change the startup.sh in the following way, what changes?
That was not the point.

Laravel + Rabbit = Crontab vs Supervisor

2 Answers