What's the difference between importing a module inside `if __name__ == '__main__'` or outside with OpenMPI?

Viewed 90

I have two executable Python3 scripts that look like the following.

script1:

#/usr/bin/env python
import sys
from mymodule import launch

if __name__ == '__main__':
    sys.exit(launch())

and

script2:

#/usr/bin/env python
import sys

if __name__ == '__main__':
    from mymodule import launch
    sys.exit(launch())

My understanding of CPython is that these two scripts should work identically.

However, if I try to run the ./script1 script using OpenMPI's mpirun like so:

$ mpirun -np 2 ./script1

it fails (on Debian linux) with the following errors:

--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  getting local rank failed
  --> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_ess_init failed
  --> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_rte_init failed
  --> Returned "No permission" (-17) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[3018108860e1:05983] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

But if I run:

$ mpirun -np 2 ./script2

it runs without error.

I wish that I could give you a self-contained, short example of the mymodule module and the go function that reproduces this error. I have tried to reduce my large production code to something small that reproduces the problem, but every time I try to trim down the production code to something small, the error is not reproducible. So, that said, I am not asking you all to tell me where my bug is.

Instead, I am wondering if any of you have any ideas as to why the two launch scripts might behave differently?

1 Answers

Importing a module under the if __name__ == "__main__" guard indicates that it's not used by any other code in the module. That is, if you import the module as a library to some other script, it'll work as intended without that import.

Some modules can take a long time to import so it is good practice to avoid importing them unless it's necessary.

Related