Supervisord : How do I execute custom php command if any of worker e.g. process id 123 stops

user_3326275 0
Viewed 2344
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php artisan queue:work --tries=3 --queue=checkin_checkout
directory=/var/www/byjus-ams
autostart=true
autorestart=true
numprocs=5
redirect_stderr=true
stdout_logfile=/var/log/supervisor/supervisord.log
#stopwaitsecs=3600

As of now my configuration for supervisor is like above which runs 5 workers in parallel. Now what I want is : When any of the workers get killed I want to fire custom php command(for eg: php artisan broadcast-process-killed). Anyone has any idea how to handle such events and fire custom php command after any worker process gets and how to configure it?

Even simpliying below

supervisor> status
laravel-worker:laravel-worker_00   RUNNING   pid 9460, uptime 4:32:43
laravel-worker:laravel-worker_01   RUNNING   pid 15907, uptime 7:01:51
laravel-worker:laravel-worker_02   RUNNING   pid 22621, uptime 6:30:17
laravel-worker:laravel-worker_03   RUNNING   pid 15909, uptime 7:01:51
laravel-worker:laravel-worker_04   RUNNING   pid 24049, uptime 6:23:00
supervisor>

If I use kill -9 9460 , I want to run custom command on. Anyone has used custom events?

laravel supervisord shell bash php
1 Answers

I found a solution where you can run a command before your actual command:

command=/bin/sh -c "php artisan broadcast-process-started; exec php artisan queue:work --tries=3 --queue=checkin_checkout"

Doing so will execute broadcast-process-started always after your worker stopped working AND restarted.

By using /bin/sh -c you make one command out of these two. By adding the execute to the second 'actual' command, supervisord will be able to handle its pid.

If you'd omit the exec, you would get the pid forked by /bin/sh and all your commands inside will spawn as subprocesses, which supervisord cant handle. So when you reload the childprocesses persist! So keep the exec!

If you flip the commands, the broadcast wont execute.

Event-Listener solution

supervisord config

[eventlistener:test-listener]
command=php /some/dir/event-listener.php <name>
events=PROCESS_STATE
stderr_logfile=/some/dir/event_err.log

Where <name> refers to [program:<name>] of your config - the program you want to observe.

event-listener.php

Written as php-script - you may code this as artisan command aswell.

<?php

declare(strict_types=1);

$programToListen = $argv[1] ?? false;
$eventsToListen = [
    'PROCESS_STATE_EXITED',
    'PROCESS_STATE_FATAL',
    'PROCESS_STATE_STOPPED',
];

if ($programToListen === false) {
    exit(1);
}

while (true) {
    // tell supervisord we are ready to receive event-headers
    echo 'READY' . PHP_EOL;
    
    $eventHeader = parseEventHeader(fgets(STDIN));
    
    if (
        ($eventHeader['groupname'] ?? '') === $programToListen &&
        in_array(($eventHeader['eventname'] ?? ''), $eventsToListen)
    ) {
        debug(sprintf(
            'Handling %s of %s',
            $eventHeader['eventname'],
            $eventHeader['groupname']
        ));
       
        // add your code here
        // exec('php artisan ...') for example
    }

    // Tell supervisord we are done handling the event-header
    echo 'RESULT 2' . PHP_EOL . 'OK';
}

function debug(string $message): void {
    fwrite(STDERR, $message . PHP_EOL);
    return;
}

function parseEventHeader(string $eventHeaderString): array {
    $keyValuePairs = explode(' ', $eventHeaderString);

    $eventArray = [];
    foreach ($keyValuePairs as $keyValuePair) {
        $keyValue = explode(':', $keyValuePair);

        $eventArray[$keyValue[0]] = $keyValue[1];
    }

    return $eventArray;
}

exit(0)
?>

trivia

  • Instead of using echo you can fwrite(STDOUT,..) aswell, but that was a little too fancy for me.
  • I left in my debug mechanism in, in case you want to fiddle around with it (you will :D).

additional information

  • Have a look here for available Process-Types to handle.
  • Have a look here for Subprocess-States (which yield the events).
  • Event-Listener notification protocol (our echo messages and event-header parsing).
  • PROCESS_STATE_EXITED will be triggered on > kill or end of execution
  • PROCESS_STATE_STOPPED will be triggered by supervisor (restart etc.)
  • I did not manage to trigger PROCESS_STATE_FATAL - but will listen to it, just in case :)
user_99684860
Related