MPI-assisted model fails to create sequential outputs

user_18421077 0
Viewed 142

I am using a numerical wave model (SWASH: https://swash.sourceforge.io/), and attempting to run simulations in parallel. Note, I am at the limits of my expertise here, so any clarification process or nomenclature are appreciated. The model uses MPICH2 to allow the user to specify the number of processes to run. In a successful parallel run on a single multi-processor machine, the model should output sequential outputs (for example a PRINT file that records information on timestepping and initialization will be output in serial, while multiple PRINT-00X files will be created in parallel, where X is the process number).

When I run the model I receive the following error for each process past the first (this first process continues successfully and outputs to the initial PRINT file):

forrtl: severe (47): write to READONLY file, unit 4, file C:\Users\DESKTOP\SWASH\vic_harbour\coarse_test\PRINT
Image              PC                Routine            Line        Source
swash.exe          00007FF70AD5C5E8  Unknown               Unknown  Unknown
swash.exe          00007FF70AD159BA  Unknown               Unknown  Unknown
swash.exe          00007FF70AC9A269  OCPINI                    290  ocpids.for
swash.exe          00007FF70A912CB9  SWASHINIT                  85  SwashInit.f90
swash.exe          00007FF70A911B69  SWASHMAIN                 119  SwashMain.f90
swash.exe          00007FF70A911AB4  MAIN__                     63  Swash.f90
swash.exe          00007FF70ADDC8C2  Unknown               Unknown  Unknown
swash.exe          00007FF70ADDCCB4  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFC67187034  Unknown               Unknown  Unknown
ntdll.dll          00007FFC67B22651  Unknown               Unknown  Unknown

If I abort the job I see error 47 reported for all but one process. Presumably, the processes are initiated successfully, but because they aren't given discrete PRINT-00X files to write to, they fail (hence the Read Only error).

I have successfully installed the program on two windows computers (win10 and 11), both using the provided installer, and by compiling the source code (including patches: https://swash.sourceforge.io/problems_and_fixes/problems_and_fixes.htm). When compiling I used the intel one API environment. I have encountered this problem using pre-defined test cases (link in above linked page) provided by SWASH folks, and tried different grid partitioning schemes (info here: https://swash.sourceforge.io/online_doc/swashimp/node10.html). MPICH is installed as follows: https://swash.sourceforge.io/online_doc/swashimp/node9.html . Unfortunately I am unable to use the mpiifort call in the free intel compiler, so cannot set things up that route, as far as I know.

Note that when compiling I receive two messages:

(1) at the nmake config step:

'uname' is not recognized as an internal or external command, operable program or batch file

But the the config file is successfully populated with windows appropriate info.

(2) at the nmake mpi step:

ifort sparskit2.for -c /O2 /assume:byterecl /traceback /nowarn /nologo /Qdiag-disable:8290 /Qdiag-disable:8291 /Qdiag-disable:8293  /include:C:\PROGRA~1\MPICH2\include

My question: I think the program is initializing the parallel process, but fails to generate the correct outputs.

Broadly, is this related to the software itself, or is there any evidence here that this is a problem with compiling/installation and/or mpi implementation.

Specifically, how would I go about getting the parallel runs to produce the appropriate outputs?

mpi numerical-methods fortran
1 Answers

The problem seems to be caused by a misconfiguration of the MPI environment. Each process requires a set of non-conflicting file descriptors for standard I/O. You are using a machine with N processors, so you require N/N+1 file descriptors for standard I/O.

To do so, you need to modify your MPICH2 setup as follows:

Set the variable MACHINEFILE in your mpirun execution script to point to a valid machinefile that only contains one line:

localhost slots=N+1

This will automatically add one extra slot for each processor on your localhost. In this case, it means that each process will be able to open N+1 file descriptors. Please note that you have to take into account possible shared memory on a single host (see FAQs). If you have multiple hosts and multiple processors on those hosts, this variable should contain the following number of slots for each host:

(total_processors_on_this_host - 1) * total_processors_on_this_host + 1 (e.g., 3 * 4 + 1 = 13)

. For instance, if you have 12 processors per node and 24 total nodes in your cluster, then this variable should look like something like this:

n0 n0 slots=12 n1 n1 slots=12 n2 n2 slots=12 n3 n3 slots=12 ... ... ...

The last line is the only one that contains the number of slots (here, 12) for each processor on the last node of your cluster. The script above will automatically add one extra slot for each processor on that node. If you have more nodes, just repeat this line for each node until you reach the end of your cluster. Once you have modified your MPICH2 setup accordingly, you should be able to run in parallel now.

You may also need to specify:

  LD_LIBRARY_PATH=/path/to/swash/lib:$LD_LIBRARY_PATH
  PATH=/path/to/swash/bin:$PATH

To make sure all these variables are available to all processes when they start up and mpirun starts them (remember, they can't see each other!). Also, please make sure that mpirun is in fact running as a service (http://superuser.com/questions/137975). You should then be able to run your mpi program as normal, but with a bit more flexibility (e.g., --byslot or --machinefile):

  mpirun -machinefile /your/hosts_file -byslot ./my_mpi_program arg1 arg2 ...

Or if you prefer not to use a machinefile at all and instead let mpich determine the processor allocation:

  mpirun -byslot ./my_mpi_program arg1 arg2 ...

Note that these options, when used, should be the first options to your mpi executable. Also, you may need to add -n option (the number of slots) to your process launcher (e.g., mpirun or srun).

user_180500830
Related