ActiveRecord not releasing memory

Viewed 955

I have an export job that exports a big amount of data from our MySQL DB. As the data grew I noticed that the sidekiq job for this takes far too much memory. The server has 32GB and after the export is done, it takes 28GB. When I stop the sidekiq process, memory use drops to 8GB.

I already followed the guide here https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting

  • prevent memory fragmentation by using MALLOC_ARENA_MAX=2
  • clear query cache ActiveRecord::Base.connection.clear_query_cache

I'm on Ruby 2.6.5p114 and tried to isolate the problem by creating a new rails app in production, and using my DB as a backend:

gem install rails --version 5.2.4.3
rails new debug -d mysql

I created an empty model to avoid custom methods in my code that maybe cause the problem:

class Variant < ApplicationRecord
end

This script simply loads 1 Mio objects from the DB and prints the memory usage:

# memory.rb

def memory
  (`ps -o rss= -p #{Process.pid}`.to_i.to_f / 1024).to_s + " MB"
end

def load_variants
  puts "load_variants..."
  Variant.uncached do
    variants = Variant.limit(1_000_000).to_a
    puts "variant.count: #{variants.count}"
  end
end

puts memory
load_variants
puts memory

puts "GC.start..."
GC.start
puts memory

# second run
load_variants
puts memory

puts "GC.start..."
GC.start
puts memory

This is the output:

root@6e79d7a97d9c:/usr/src/debug# rails r memory.rb
76.93359375 MB
load_variants...
variant.count: 1000000
2436.3125 MB
GC.start...
2421.046875 MB
load_variants...
variant.count: 1000000
2436.3828125 MB
GC.start...
2436.3984375 MB
  1. it starts with 76.93359375 MB
  2. after loading 1 Mio objects, memory increases to 2436.3125 MB
  3. garbage collection reduces memory to 2421.046875 MB, but I would expect a significantly higher drop!
  4. interestingly, a second run, only increases memory to 2436.3828125 MB
  5. the last GC.start somehow increases the memory a little bit to 2436.3984375 MB

So I'd like to know how this could be? There must be something in ActiveRecord that I'm unaware, and I'd like to understand how this all works, and why the memory isn't freed.

Following this logic, the memory should increase on every request that reads data, but I assume there is something different when using within a request-response cycle.

1 Answers

Loading large object that are spread all around the memory (instead of objects like String that are in consecutive memory) in Ruby's tend to have such effect since the Mark & Sweep algorithm cannot return the whole blocks of memory back to the OS. You'll get similar effect if you start parsing large JSON files (like 10+MB) since the resulting Hash, consisting of tons of other objects, is going to be placed in multiple memory blocks along side other objects that still have an active reference and hence Ruby cannot release that block.

Related