Python unittest C/Fortran MPI functions resulting in MPI_Init function getting called after MPI_finalize

Viewed 144

I am writing Python wrappers to a Fortran library which uses MPI (OpenMPI, if that's relevant). The problem I mention here also occurs exactly for Python wrappers to C libraries. I must use the inbuilt unittest Python package for testing the Python interface.

I have the following MWE which tries to find maximum and minimum integer across all processors. Here's the Fortran file:

! File mpitest.F90
module mpitest
    implicit none
    include "mpif.h"
    integer nranks
    integer rank

    contains

    !--------------------------------------------------!
    subroutine pympitest_init()
        implicit none
        integer ierror
        call MPI_INIT(ierror)
        call MPI_COMM_SIZE(MPI_COMM_WORLD, nranks, ierror)
        call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
    end subroutine pympitest_init

    !--------------------------------------------------!
    subroutine get_max_integer(inint, outint)
        implicit none
        integer, intent(in) :: inint
        integer, intent(out) :: outint
        integer ierror
        call mpi_allreduce(inint,outint,1,mpi_integer,mpi_max,MPI_COMM_WORLD,ierror)
    end subroutine get_max_integer

    !--------------------------------------------------!
    subroutine get_min_integer(inint, outint)
        implicit none
        integer, intent(in) :: inint
        integer, intent(out) :: outint
        integer ierror
        call mpi_allreduce(inint,outint,1,mpi_integer,mpi_min,MPI_COMM_WORLD,ierror)
    end subroutine get_min_integer

    !--------------------------------------------------!
    subroutine pympitest_final()
        integer ierror
        call mpi_finalize(ierror)
    end subroutine pympitest_final

end module mpitest

The Python unit test file is given below, with some things that I have tried but don't help put in as comments:

#!/usr/bin/env python3
# Python file test_pympitest.py
from __future__ import print_function
import unittest
import numpy as np

from pympitest import mpitest as pm

#--------------------------------------------------------------------#
class Test_Fortran_MPI_functions(unittest.TestCase):
    """Unit test template for testing MPI functions."""

    #----------------------------------------------------------------#
    #def __init__(self):
    #    """Try MPI_Initialize -- Does not help."""
    #    print("Initializing MPI")
    #    pm.pympitest_init()

    #----------------------------------------------------------------#
    #def tearDown(self):
    #    """Try MPI_Finalize -- Does not help."""
    #    print("Finalizing MPI")
    #    pm.pympitest_final()

    #----------------------------------------------------------------#
    def setUp(self):
        """Initialize the unit test."""
        print("Initializing MPI")
        pm.pympitest_init()

        # We'll just test min. and max. of ranks across all PEs.
        self.inint = pm.rank

    #----------------------------------------------------------------#
    def tearDown(self):
        """Finalize the unit test."""
        print("Finalizing MPI")
        pm.pympitest_final()

    #----------------------------------------------------------------#
    #----------------------------------------------------------------#
    # Two Tests. If you comment out any one of the tests, then the
    # unit test runs fine. Keeping both uncommented thorws error
    # saying MPI_Init was called after MPI_Final, obviously since
    # the setUp and tearDown functions are called every time.
    #----------------------------------------------------------------#
    def test_get_min_integer(self):
        """Test get_min_integer function."""
        minint = pm.get_min_integer(self.inint)
        print("Minimum on processor {} = {}".format(pm.rank,minint))
        self.assertEqual(minint, 0)

    #----------------------------------------------------------------#
    def test_get_max_integer(self):
        """Test get_max_integer function."""
        maxint = pm.get_max_integer(self.inint)
        print("Maximum on processor {} = {}".format(pm.rank,maxint))
        self.assertEqual(maxint, pm.nranks-1)

#--------------------------------------------------------------------#
if __name__ == "__main__":
    unittest.main()

    # Tried the following. Does not help:
    # Calls MPI_Initialize but exits without calling MPI_Finalize.
    #print("Initializing MPI")
    #pm.pympitest_init()
    #unittest.main()
    #print("Finalizing MPI")
    #pm.pympitest_final()

The Fortran code is wrapped into Python module called pympitest using f2py, like so: f2py --f90exec="mpif90" -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -c mpitest.F90 -m pympitest. The unit test is run on 4 processors like so: mpirun -np 4 python test_pympitest.py.

I want to keep all MPI unit tests in a single python file. In the different things I have tried out (see the comments in the Python file), the problem I have faced is that MPI_Initialize and MPI_Finalize either get called multiple times in a single run or only one of them gets called, all of which exit with an error.

How do I fix this so that the unit tests pass properly?


Edit (Addition): Putting the testing functions into separate Test_...(unittest.TestCase) classes within a single Python file also does not help. Splitting the MPI unit tests into different files and trying python -m unittest discover does not work, which is inconvenient. Testing a single wrapped MPI function per Python file and individually running the each Python file would work, but that is something I would rather avoid.

1 Answers

The setUp() and tearDown() methods of a unittest.TestCase provide per-test-method setup and teardown. You should expect that the former is executed before each test method and the latter is executed after each.

You seem instead to want those to be executed once each for the whole collection of test methods in the class. For that you want setUpClass() and tearDownClass(). Alternatively, if you define multiple test classes then you may prefer setUpModule() and tearDownModule().

Alternatively, you could create a unittest.TestSuite subclass with which to run your tests, and override its run() method to perform once-for-all setup and teardown.

Related