Optimize ram usage Laravel, split Cronjobs into chunks?

Viewed 375

I have a Laravel backend which is fetching data from various APIs every day and processing them. There is one command for fetching those data and storing them in the database and one command which is using these data to update wordpress fields. This is because laravel is just the backend for my wordpress site and laravel takes care of updating fields of my online shop.

At the moment there are 16.000 products online. So the command which uses the fetched and stored data from the database to update fields in wordpress has to go through 16.000 products in order to update the fields of each product. But here comes now the problem. I have a ram limit of 128Mb by my hoster. But to finish the task of updating the fields of wordpress I need about 280Mb. You can see this here on the figure (x-axias products, y-axias ram usage):

enter image description here

The blue line represents a script where I fetch all the fields and products to update via a raw SQL statement and the orange represents a script where I use a laravel model to process the fields I want to update. Everything besides the SQL statement and the laravel model is the same!

As you can see the SQL statements starts off with less MB but is not as efficient with a large amount of products as the version which is using a laravel model to process the data. The script which is using the laravel model is slightly better.

The version which is using the laravel model to process the data is using the laravel chunk method to split up the models. I guess this is also the answer to the small gaps on the orange line.

My question now is if there is the possibility to chunk the products I want to process before the script gets executed? In this way the garbage collector can remove the data from the script before which has processed a chunk of data and the ram gets cleaned.

Is there a possibility to set up a cron which is processing 0-5000 products, the next one 5000-10000 products and so on and so on. If so, how can I implement this? Will the garbage collector also remove all variables from the ram which where used in the script before?

I will also paste you my code below here but I don't think its really helpful. Any ideas are welcome to minimize the ram usage!

Code of the script which is using laravel models:

public function handle(UpdateFields $updateFields, UpdateStatus $updateStatus, UpdateTerms $updateTerms)
{
  $count = 0;
  $chunkSize = 5000;
  $countProducts = Post::lastProductId();

  while($count < $countProducts){
      $products = Post::allProductsLiveWithMetas($count, $chunkSize);
      
      // Loop trough each product
      foreach($products as $product){
          // Update fields
          $updatedFields = $updateFields->setProduct($product)->update();

          // Update Status
          $status = $updateStatus->setProduct($product)->update();

          // Update Terms
          // Update Gender
          $gender = $updateTerms->setProduct($product)->set_gender();

          // Update Size
          $size = $updateTerms->setProduct($product)->set_size();
          
          // Update Sale
          $sale = $updateTerms->setProduct($product)->set_sale();

          // Delete Wordpress Cache to save from memory leakage
          wp_cache_delete($product->ID, 'post_meta');

          unset($product);
          unset($updatedFields);
          unset($status);
          unset($gender);
          unset($size);
          unset($sale);
      }

      unset($products);
      $count = $count + $chunkSize;
  }
}

Kind regards

2 Answers

Two approaches come to mind...

  1. Chain the requests, pass in the current offset, augment, when script finishes do a redirect and call again with new offset. Your CRON invokes the script with offset 0 to start the chain of execution. Stop when you reach the upper bound.

  2. Write a simple Bash script and have the CRON fire it, running your PHP scripts.

#/bin/bash
php /srv/somepath/script.php 0 500
php /srv/somepath/script.php 500 500
php /srv/somepath/script.php 1000 500

# or wget http://example.com/script.php?offset=0&limit=500

Implementing Queueing system with Redis can resolve your memory problem and also will give you better control over your implementation of fetching data from external sources and updating the same in Database. In the queue, you are going to deal with fewer data and one by one with retry option to make sure all the operations get completed without fail.

For your reference

Laravel queue documentation

Related