(Laravel 5) Monitor and optionally cancel an ALREADY RUNNING job on queue

user_697579 0
Viewed 1694

I need to achieve the ability to monitor and be able to cancel an ALREADY RUNNING job on queue.

There's a lot of answers about deleting QUEUED jobs, but not on an already running one.

This is the situation: I have a "job", which consists of HUNDREDS OF THOUSANDS rows on a database, that need to be queried ONE BY ONE against a web service.

Every row needs to be picked up, queried against a web service, stored the response and its status updated.

I had that already working as a Command (launching from / outputting to console), but now I need to implement queues in order to allow piling up more jobs from more users.

So far I've seen Horizon (which doesn't runs on Windows due to missing process control libs). However, in some demos seen around it lacks (I believe) a couple things I need:

  • Dynamically configurable timeout (the whole job may take more than 12 hours, depending on the number of rows to process on the selected job)
  • Ability to CANCEL an ALREADY RUNNING job.

I also considered the option to generate EACH REQUEST as a new job instead of seeing a "job" as the whole collection of rows (this would overcome the timeout thing), but that would give me a Horizon "pending jobs" list of hundreds of thousands of records per job, and that would kill the browser (I know Redis can handle this without itching at all). Further, I guess is not possible to cancel "all jobs belonging to X tag".

I've been thinking about hitting an API route, fire the job and decouple it from the app, but I'm seeing that this requires forking processes.

For the ability to cancel, I would implement a database with job_id, and when the user hits an API to cancel a job, I'd mark it as "halted". On every loop I would check its status and if it finds "halted" then kill itself.

If I've missed any aspect just holler and I'll add it or clarify about it.

So I'm asking for an advice here since I'm new to Laravel: how could I achieve this?

laravel-5.8 queue
2 Answers

So I finally came up with this (a bit clunky) solution:

In Controller:

 public function cancelJob()
    {
        $jobs = DB::table('jobs')->get();

        # I could use a specific ID and user owner filter, etc.
        foreach ($jobs as $job) {
            DB::table('jobs')->delete($job->id);
        }

        # This is a file that... well, it's self explaining
        touch(base_path(config('files.halt_process_signal')));

        return "Job cancelled - It will stop soon";
    }

In job class (inside model::chunk() function)

                # CHECK FOR HALT SIGNAL AND [OPTIONALLY] STOP THE PROCESS
                if ($this->service->shouldHaltProcess()) {

                    # build stats, do some cleanup, log, etc...

                    $this->halted = true;
                    $this->service->stopProcess();

                    # This FALSE is what it makes the chunk() method to stop looping
                    return false;
                }

In service class:

    /**
     * Checks the existence of the 'Halt Process Signal' file
     *
     * @return bool
     */
    public function shouldHaltProcess() :bool
    {
        return file_exists($this->config['files.halt_process_signal']);
    }


    /**
     * Stop the batch process
     *
     * @return void
     */
    public function stopProcess() :void
    {
        logger()->info("=== HALT PROCESS SIGNAL FOUND - STOPPING THE PROCESS ===");
        $this->deleteHaltProcessSignalFile();

        return ;
    }

It doesn't looks quite elegant, but it works. I've surfed the whole web and many goes for Horizon or other tools that doesn't fit my case. If anyone has a better way to achieve this, it's welcome to share.

user_6975790

Laravel queue have 3 important config: 

1. retry_after
2. timeout
3. tries

See more: https://laravel.com/docs/5.8/queues

Dynamically configurable timeout (the whole job may take more than 12 hours, depending on the number of rows to process on the selected job)

I think you can config timeout + retry_after about 24h.

Ability to CANCEL an ALREADY RUNNING job.

  • Delete job in jobs table
  • Delete process by process id in your server

Hope it help you :)

user_37138460
Related