Laravel Horizon queue keeps pausing for one minute

Question

TL;DR Laravel Horizon queue workers go to sleep for 60 seconds after each job they process

I have a big backlog in my Laravel Horizon queue. There are a lot of workers (maxProcesses set to 30), but when I monitor the log file, the output suggests that it is processing exactly 30 jobs over the course of 2-3 seconds, and then it pauses for a full minute (more or less exactly 60 seconds).

Any ideas why this could be happening? Am I hitting some resource limit that is causing Horizon or Supervisor to hit the breaks?

Here's the relevant section from my horizon.php config file:

'environments' => [
        'production' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['high', 'default', 'low'],
                'balance' => 'false',
                'minProcesses' => 3,
                'maxProcesses' => 30,
                'timeout' => 1800,
                'tries' => 3
            ],

I have the exact same configuration in my local environment, and my throughput locally is ~600 jobs/minute. In production it hovers right around ~30 jobs/minute.

Update per @Qumber's request

For the most part these aren't actually jobs. They're events being handled by one or more listeners, most of which are super simple. For example:


public function handle(TransactionDeleted $event)
{
    TransactionFile::where("transaction_id", $event->subject->id)->delete();
}

Here's some queue config:

'redis' => [
    'driver' => 'redis',
    'connection' => 'default',
    'queue' => env('REDIS_QUEUE', 'default'),
    'retry_after' => 1900,
    'block_for' => null,
],

Update per @sykez request

Here's the supervisor config in local:

[program:laravelqueue]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/artisan queue:once redis --sleep=1 --tries=1

autostart=true
autorestart=true
user=adam
numprocs=3
redirect_stderr=true
stdout_logfile=/path/to/worker.log
stopwaitsecs=3600

Here's the supervisor config in production:

[program:daemon-XXXXXX]
directory=/home/forge/SITE_URL/current/
command=php artisan horizon

process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
user=forge
redirect_stderr=true
stdout_logfile=/home/forge/.forge/daemon-XXXXXX.log
stopwaitsecs=3600

The local supervisor is running the queue directly, with the "once" flag, which should load the entire code base for each job rather than running as a daemon. This, of course, should make it slower, not 20 times faster...

Another update Thanks to some help from one of the core Laravel devs, we were able to determine that all of the "hanging" jobs were broadcast jobs, from events that were configured to broadcast after firing. We use Pusher as our broadcast engine. When Pusher is disabled (as it is in our local environment), then the jobs finish immediately with no pause.

Laravel Horizon queue keeps pausing for one minute

0 Answers