Laravel Bus Batch pending jobs is negative

Question

I'm trying to load shopify products through cache and im using Redis::throttle to prevent rate limit in shopify. My problem is I put all the process of fetching in a Batch and it returns a negative no. of pending jobs.

Here's the batch info.

and here's my approach

public function load(Request $request, Sample $sample) {

    return DB::transaction(function () use ($sample) {
        $batch = Bus::batch([])->dispatch();

            $since_id = 0;

            while ($since_id >= 0) {
                $fetchedProducts = (new ShopifyProduct(
                    $sample->shopify_domain,
                    $sample->shopify_info['access_token'])
                )
                ->getById($since_id)
                ->then(function ($data) {
                    return $data['products'];
                }, function () {
                    return [];
                })
                ->wait();

                if (collect($fetchedProducts)->count() == 0) break;

                $lastProduct = Arr::last($fetchedProducts);
                $since_id = $lastProduct['id'];

                collect($fetchedProducts)
                    ->each(function ($shopifyProduct) use($merchant, &$batch) {
                        $batch->add(new CacheProducts($shopifyProduct, $sample));
                    });
            }
        

        return $this->okResponse(['batch_id' => $batch->id])
            ->header('Content-Type', 'application/vnd.api+json');
    });
}

and here's the job

/**
 * Execute the job.
 *
 * @return void
 */
public function handle()
{
    Redis::throttle("shopify-cache")->allow(30)->every(60)->block(70)->then(function () {
        (new Metafield(
                $this->merchant->shopify_domain,
                $this->merchant->shopify_info['access_token']
            )
        )
        ->get($this->shopifyProduct['id'], 'products')
        ->then(function ($data) {
            $this->shopifyProduct['metafields'] = $data['metafields'];

            $cacheProducts = Cache::tags($this->merchant->name)->get('products') ?? [];

            array_push($cacheProducts, $this->shopifyProduct);

            Cache::tags($this->merchant->name)->put('products', $cacheProducts, now()->addHour());

        }, function ($e) {
        })
        ->wait();
    }, function () {
        return $this->release(10);
    });
}

user_6385459 · Answer

You need to refactor you code which is dispatching the jobs for better understanding what exactly you are doing and to simplify your code.

Besides of that, you are dispatching an empty batch and later in the loop you are adding jobs to the batch with $batch->add() which is totally fine, but you are passing the &$batch variable in your closure, which, I think is not working as you expect.

And what is $fetchedProducts returning? you are fetching it by ID, but the lastId is the id of the last item in that array, isn't that the same id then?

I think it will help you if you split your code in peaces and debug every part to check if the job batches are working properly:

First make sure your job's handle is doing nothing but directly returning (return;) so you can first test your load class.

public function handle()
{
  return;

  // All of the original code in this method you can comment out, for quicker testing.
}

Just cut your load() method in pieces, so you can step by step check if every line of code is working as you expect.

Refactor your code to better understand what is happening. I've made some changes in the hope this code will be somewhat more readable;

 return DB::transaction(function () use ($sample) {
     $jobs = [];

     // Fetch all products (if there are too many products, you may use paginated results)
     $fetchedProducts = (new ShopifyProduct(
         $sample->shopify_domain,
         $sample->shopify_info['access_token'])
     )
     // Can you explain how $since_id can be different for each product if you are fetching products by that same id?
     //->getById($since_id)
     ->then(function ($data) {
         return $data['products'];
     }, function () {
         return [];
     })->wait();

     if (collect($fetchedProducts)->empty()) {
         return $this->okResponse(['message' => 'No products to process.'])
             ->header('Content-Type', 'application/vnd.api+json');
     }

     collect($fetchedProducts)
         ->each(function ($shopifyProduct) use($merchant, &$jobs) {
             $jobs[] = new CacheProducts($shopifyProduct, $sample)
         });

     // After all jobs are added, we can dispatch them.
     $batch = Bus::batch($jobs)->dispatch();

     return $this->okResponse(['batch_id' => $batch->id])
         ->header('Content-Type', 'application/vnd.api+json');
 });

Laravel Bus Batch pending jobs is negative

1 Answers