Simulating many agents in PyTorch using multiprocessing

user_9220683 0
Viewed 270

I want to simulate multiple reinforcement learning agents that are coded using Pytorch. The agents do not share any data dynamically, so I expect that the task should be "embarassingly parallel". I need a lot of simulations (I want to see what is the distribution my agents converge to) so I hope to speed it up using multiprocessing.

I have a model class that stores all the parameters of my agents (which are the same across agents) and the environment. I can simulate N agents over T periods using

model.simulate(N = 10, T = 50)

My class would then run simulation loops and store all networks and simulation histories. I am very new to parallel programming, and I (naively) try the following:

import torch.multiprocessing as mp

num_processes = 6
processes = []
for _ in range(num_processes):
    p = mp.Process(target=model.simulate(N = 10, T = 50), args= ())
    p.start()
    processes.append(p)
for p in processes:
    p.join()

For now I do not even try to store results, I just want to see some speed-up. But the time it takes to run the code above is roughly the same as when I simply run a loop and do 6 simulations consequently:

for _ in range(num_processes):
    model.simulate(N = 10, T = 50)

I also tried to make processes for different instances of the model class, but it did not help.

reinforcement-learning pytorch montecarlo multiprocessing
1 Answers

It looks like your problem is in this line

p = mp.Process(target=model.simulate(N = 10, T = 50), args= ())

The part model.simulate(N = 10, T = 50) is executed first, then the result (I'm assuming None if there is no return from this method) is passed to the mp.Process as the target parameter. So you are doing all the computation sequentially, and not performing it on the new processes.

What you need to do instead is to pass the simulate function (without executing it) and provide the args separately.

i.e. something like...

p = mp.Process(target=model.simulate, args=(10, 50))

Providing target=model.simulate will pass a reference to the function itself rather than executing it and passing the result. This way it will be executed on the new process and you should acheive the parallelism.

See offical docs for an example.

user_71681860
Related