Laravel 8 chunk memory leak

Viewed 37

I have an artisan command that is scheduler to run every 15 minutes to update a model calculated field. The command process thousands of rows and ends up eating all of the memory.

My understanding of the laravel documentation is that using the chunk method reduces greatly the memory usage when iterating over lots of rows because it only keeps N number of rows in memory at a time instead of the whole dataset. This is the method I am already using, but every additional processed chunks increase the memory consumption of the docker container.

Here is the original code:

public function __construct()
{
    parent::__construct();
    $this->openingHourArraySerialiser = new OpeningHourArraySerializer();
}

public function handle()
{
    $this->openingHours = OpeningHour::with('scheduleHours')->get();
    ini_set('memory_limit', '1024M');
    $this->getTicketQuery()->chunk(500, function($tickets) {
        $cases = [];
        $ids = [];
        $params = [];
        foreach($tickets as $ticket) {
            $cases[] = "WHEN {$ticket->ID} then ?";
            $openingHourSchedule = $this->getOpeningHoursSchedule($ticket);
            $params[] = TicketPriorityLevelCalculator::calculate($ticket, $openingHourSchedule);
            $ids[] = $ticket->ID;
        }
        $ids = implode(',', $ids);
        $cases = implode(' ', $cases);
        if (!empty($ids)) {
            DB::connection('tcollect')->update("UPDATE _DSKTickets SET PriorityLevel = CASE ID {$cases} END WHERE ID in ({$ids})", $params);
        }
    });
    $this->setTicketWithoutDueDatePriority();
}

private function getOpeningHoursSchedule($ticket)
{
    $openingHours = $this->openingHours->where('ID', $ticket->OpeningHoursID)->first();
    return $openingHours->getSerializedSchedule($this->openingHourArraySerialiser);
}

I have read from other thread suggesting to disable the query log. I have also set the APP_DEBUG to false.

The following image is a graph of the memory consumption before and after.

It reached 1gb and dies (I manually killed the process at 500mb for faster testing of my modifications)

Memory usage graph

I have also tried:

  • to use the chunkById method.
  • to use lazy and lazyById methods
  • unset cases, ids, params and tickets variables

It result in the same memory consumption. I cannot use the cursor method because I need to eager load relations.

How can I reduce the memory usage to never reach the 1gb limit?

0 Answers
Related