Multithreading in Rails: Circular dependency detected while autoloading constant

Question

I have a Rails app in which I have a Rake task that uses multithreading functions supplied by the concurrent-ruby gem.

From time to time I encounter Circular dependency detected while autoloading constant errors.

After Googling for a bit I found this to be related to using threading in combination with loading Rails constants.

I stumbled upon the following GitHub issues: https://github.com/ruby-concurrency/concurrent-ruby/issues/585 and https://github.com/rails/rails/issues/26847

As explained here you need to wrap any code that is called from a new thread in a Rails.application.reloader.wrap do or Rails.application.executor.wrap do block, which is what I did. However, this leads to deadlock.

The recommendation is then to use ActiveSupport::Dependencies.interlock.permit_concurrent_loads to wrap another blocking call on the main thread. However, I am unsure which code I should wrap with this.

Here's what I tried, however this still leads to a deadlock:

@beanstalk = Beaneater.new("#{ENV.fetch("HOST", "host")}:#{ENV.fetch("BEANSTALK_PORT", "11300")}")
tube_name = ENV.fetch("BEANSTALK_QUEUE_NAME", "queue")

pool = Concurrent::FixedThreadPool.new(Concurrent.processor_count * 2)

# Process jobs from tube, the body of this block gets executed on each message received
@beanstalk.jobs.register(tube_name) do |job|
    ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
      @logger.info "Received job: #{job.id}"
      Concurrent::Future.execute(executor: pool) do
        Rails.application.reloader.wrap do
          # Stuff that references Rails constants etc
          process_beanstalk_message(job.body)
        end
      end
    end
end

@beanstalk.jobs.process!(reserve_timeout: 10)

Can anyone shed a light as to how I should solve this? The odd thing is I encounter this in production while other information on this topic seems to imply it should normally only occur in development.

In production I use the following settings:

config.eager_load = true

config.cache_classes = true.

Autoload paths for all environments are Rails default plus two specific folders ("models/validators" & "jobs/concerns").

eager_load_paths is not modified or set in any of my configs so must be equal to the Rails default.

I am using Rails 5 so enable_dependency_loading should equal to false in production.

user_5135939 · Answer

I had this issue while trying out two gems that handles parallel processing;

pmap gem
parallel gem

For pmap I kept getting an error related to Celluloid::TaskTerminated and for parallel I was getting a Circular dependency detected while autoloading constant for when I ran it with more than 1 thread. I knew this issue was related to how my classes and modules were eager loading and race to be placed on a thread. I try enabling both of the configs to true config.cache_classes = true and config.eager_load = true in the development env and that did the trick for me.

Multithreading in Rails: Circular dependency detected while autoloading constant

3 Answers