I have two executable Python3 scripts that look like the following.
script1:
#/usr/bin/env python
import sys
from mymodule import launch
if __name__ == '__main__':
sys.exit(launch())
and
script2:
#/usr/bin/env python
import sys
if __name__ == '__main__':
from mymodule import launch
sys.exit(launch())
My understanding of CPython is that these two scripts should work identically.
However, if I try to run the ./script1 script using OpenMPI's mpirun like so:
$ mpirun -np 2 ./script1
it fails (on Debian linux) with the following errors:
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
getting local rank failed
--> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
orte_ess_init failed
--> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):
ompi_mpi_init: ompi_rte_init failed
--> Returned "No permission" (-17) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
[3018108860e1:05983] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
But if I run:
$ mpirun -np 2 ./script2
it runs without error.
I wish that I could give you a self-contained, short example of the mymodule module and the go function that reproduces this error. I have tried to reduce my large production code to something small that reproduces the problem, but every time I try to trim down the production code to something small, the error is not reproducible. So, that said, I am not asking you all to tell me where my bug is.
Instead, I am wondering if any of you have any ideas as to why the two launch scripts might behave differently?