Understanding Put/Get in mpi4py

user_8516804 0
Viewed 498

I'm learning how to use one-sided communication in mpi4py. To test my understanding, I came up with this contrived example:

import mpi4py.MPI as mpi
import numpy as np
import time

def main():
    rank = mpi.COMM_WORLD.Get_rank()
    n_proc = mpi.COMM_WORLD.Get_size()

    assert(n_proc == 2)

    buff = np.zeros(10, dtype='d')
    win = mpi.Win.Create(buff, 1, mpi.INFO_NULL, mpi.COMM_WORLD)

    if (rank == 0):
        win.Lock(1)
        win.Put([buff, mpi.DOUBLE], 1)
        win.Unlock(1)

        buff[-1] = 9
        time.sleep(30)

        win.Lock(1)
        win.Put([buff, mpi.DOUBLE], 1)
        win.Unlock(1)

    else:
        holder = np.zeros(10)
        failures = 0
        while (holder[-1] != 9):
            failures += 1
            win.Lock(0)
            win.Get([holder, mpi.DOUBLE], 0)
            win.Unlock(0)
        print('Took', failures, 'dials')

if __name__=='__main__':
    main()

I expect this to work as follows: rank 0 will wait for 30 seconds before placing the ndarray containing [0, 0, ..., 0, 9] into the shared memory. Meanwhile, rank 1 will continuously check to see if rank 0 has updated its shared memory---once it has, it will print out what it sees. Specifically, I would expect that rank 1 would execute its while loop many many times during the 30 second wait while rank 0 is idle. However, when I run this code with mpirun -n 2 python mwe.py, I consistently get this output:

Took 2 dials

indicating that the call to Get only happened twice.

Specific question: Why are there only two calls to Get from rank 1?

More vague question: Am I making an obvious mistakes in this MWE? I am pretty new to mpi4py/MPI in general.

mpi4py mpi parallel-processing python-3.x
1 Answers

You are creating as you said correctly a shared memory window. The window grants other MPI processes in the communicator direct access to the exact memory where your buff is stored. When you change buff[-1] = 9 this is ofcourse stored in said memory and the window is immediatly affected. Thus, the explicit Put call to the local process (from rank 0 to rank 0) is unnecessary and Put is useful only to place data in remote memory windows. The other MPI processes can start reading out your changed buff variable from the window before rank 0 even begins its 30 second sleep operation.

Unfortunately the reason why you see "Took 2 dials" is because rank 0 and 1 execute their first lock and unlock probably at the same time, then rank 0 executes the buff[-1] = 9 and then the loop repeats on rank 1 and it reads the updated buffer and exits after 2 iterations.

PS: To test, place a 1s sleep before buff[-1] = 9

user_10160040
Related