I am using Keras with TensorFlow backend in a cluster environment.
I'm not doing any back-propagation on the network but only using predict(...), layer.get_weights() and layer.set_weights(...).
I am manipulating the parameters of the neural network directly, optimizing a fitness function using evolution strategies.
I have a master, generating N parameter sets (weights and bias) which are then passed to workers using MPI. For this, I use mpi4py. Those workers then set those parameters on their own instance of a sequential model (keras) and evaluate the fitness function by predicting actions utilizing this network. The fitness is then transferred back to the master which adapts the distribution from which the parameter sets are sampled.
When running my code, I get the following warning:
An MPI process has executed an operation involving a call to the
fork()system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your MPI job may hang, crash, or produce silent data corruption. The use offork()(orsystem()or other calls that create child processes) is strongly discouraged.
I implemented a version of my algorithm without keras/tensorflow, and the error was gone. I, therefore, believe that tensorflow is doing the system() or fork() call. Is it possible to prevent tensorflow from doing such calls?