Export large data using WhereIn and cursor in Laravel

Viewed 20

I have a collection of 25k rows that I get from a CSV file, which contains a URL column.

How can I perform an optimized query using the value of the URL column?

This is what i have so far:

public function export()
{
   $collection = FastExcel::import($file);
   $urls = $this->getData($collection);
   FastExcel::data($urls)->export('file.xlsx');
}

// generator function
public function getData( $collection )
{
   $query = MyModel::select('url', 'views')
      ->whereIn('url', $collection->pluck('url'))
      ->cursor();

   foreach ($query as $row) {
      yield $row;
   }
}

This works using little memory, but the query time is very high.

Memory usage    54 MB
Query: select `url`, `views` from `my_table` where `url` in...  297.36ms    

I am using Laravel 9, MySQL 5.7

1 Answers

Build your query with query builder ( db raw ) to get one result , then do your magic

Related