I have an export job that exports a big amount of data from our MySQL DB. As the data grew I noticed that the sidekiq job for this takes far too much memory. The server has 32GB and after the export is done, it takes 28GB. When I stop the sidekiq process, memory use drops to 8GB.
I already followed the guide here https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting
- prevent memory fragmentation by using
MALLOC_ARENA_MAX=2 - clear query cache
ActiveRecord::Base.connection.clear_query_cache
I'm on Ruby 2.6.5p114 and tried to isolate the problem by creating a new rails app in production, and using my DB as a backend:
gem install rails --version 5.2.4.3
rails new debug -d mysql
I created an empty model to avoid custom methods in my code that maybe cause the problem:
class Variant < ApplicationRecord
end
This script simply loads 1 Mio objects from the DB and prints the memory usage:
# memory.rb
def memory
(`ps -o rss= -p #{Process.pid}`.to_i.to_f / 1024).to_s + " MB"
end
def load_variants
puts "load_variants..."
Variant.uncached do
variants = Variant.limit(1_000_000).to_a
puts "variant.count: #{variants.count}"
end
end
puts memory
load_variants
puts memory
puts "GC.start..."
GC.start
puts memory
# second run
load_variants
puts memory
puts "GC.start..."
GC.start
puts memory
This is the output:
root@6e79d7a97d9c:/usr/src/debug# rails r memory.rb
76.93359375 MB
load_variants...
variant.count: 1000000
2436.3125 MB
GC.start...
2421.046875 MB
load_variants...
variant.count: 1000000
2436.3828125 MB
GC.start...
2436.3984375 MB
- it starts with
76.93359375 MB - after loading 1 Mio objects, memory increases to
2436.3125 MB - garbage collection reduces memory to
2421.046875 MB, but I would expect a significantly higher drop! - interestingly, a second run, only increases memory to
2436.3828125 MB - the last
GC.startsomehow increases the memory a little bit to2436.3984375 MB
So I'd like to know how this could be? There must be something in ActiveRecord that I'm unaware, and I'd like to understand how this all works, and why the memory isn't freed.
Following this logic, the memory should increase on every request that reads data, but I assume there is something different when using within a request-response cycle.