How to clear memory for a long running rake task to prevent exceeding Heroku's memory quota?

user_764955 0
Viewed 1246

I have a rake task that I need to run on Heroku in the background as a one off task. However the tasks is pretty large and I run into a 'Error R14 (Memory Quota Exceeded)` and was hoping I could get some tips on how I can avoid this.

Essentially the task looks at the Products table and find products that have no images Product.where(images: nil). The task then cycles through each entry; using product.url it opens a connection to a remote website (using Nokogiri) and pulls the images and some additional data. The images are resized using mini_magick and saved to an S3 Bucket using carrierwave.

I have about 39000 records that need processing, but after about 500 I get the Memory Quota Exceeded error and the task stops.

I can see why this is quite a memory intensive task, but I was wondering if anyone could point me in the right direction as to how I could clean up the memory after each record has been processed and saved (or even after every 100 records).

Alternatively/additionally is there a way to auto restart the Heroku task after it automatically terminates?

rake-task out-of-memory heroku memory-leaks ruby-on-rails
1 Answers

Is you iterate over each record, you could force the GC to start:

Products.where(images: nil).each_with_index do |image, index|
  if index % 100 == 0
    GC.start
  end
end
user_12363360
Related