Jay_1891
0
I have an MPI program which compiles and runs, but I would like to step through it to make sure nothing bizarre is happening. Ideally, I would like a simple way to attach GDB to any particular process, but I'm not really sure whether that's possible or how to do it. An alternative would be having each process write debug output to a separate log file, but this doesn't really give the same freedom as a debugger.
Are there better approaches? How do you debug MPI programs?
I have found gdb quite useful. I use it as
mpirun -np <NP> xterm -e gdb ./program
This the launches xterm windows in which I can do
run <arg1> <arg2> ... <argN>
usually works fine
You can also package these commands together using:
mpirun -n <NP> xterm -hold -e gdb -ex run --args ./program [arg1] [arg2] [...]
As someone else said, TotalView is the standard for this. But it will cost you an arm and a leg.
The OpenMPI site has a great FAQ on MPI debugging. Item #6 in the FAQ describes how to attach GDB to MPI processes. Read the whole thing, there are some great tips.
If you find that you have far too many processes to keep track of, though, check out Stack Trace Analysis Tool (STAT). We use this at Livermore to collect stack traces from potentially hundreds of thousands of running processes and to represent them intelligently to users. It's not a full-featured debugger (a full-featured debugger would never scale to 208k cores), but it will tell you which groups of processes are doing the same thing. You can then step through a representative from each group in a standard debugger.
http://github.com/jimktrains/pgdb/tree/master is a utility I wrote to do this very thing. There are some docs and feel free to pm me for questions.
You basically call a perl program that wraps GDB and funnels it's IO to a central server. This allows GDB to be running on each host and for you to access it on each host at the terminal.
The "standard" way to debug MPI programs is by using a debugger which supports that execution model.
On UNIX, TotalView is said to have good suppoort for MPI.
I use this little homebrewn method to attach debugger to MPI processes - call the following function, DebugWait(), right after MPI_Init() in your code. Now while the processes are waiting for keyboard input, you have all the time to attach the debugger to them and add breakpoints. When you are done, provide a single character input and you are ready to go.
static void DebugWait(int rank) {
char a;
if(rank == 0) {
scanf("%c", &a);
printf("%d: Starting now\n", rank);
}
MPI_Bcast(&a, 1, MPI_BYTE, 0, MPI_COMM_WORLD);
printf("%d: Starting now\n", rank);
}
Of course you would want to compile this function for debug builds only.
Quite a simple way to debug an MPI program.
In main () function add sleep (some_seconds)
Run the program as usual
$ mpirun -np <num_of_proc> <prog> <prog_args>
Program will start and get into the sleep.
So you will have some seconds to find you processes by ps, run gdb and attach to them.
If you use some editor like QtCreator you can use
Debug->Start debugging->Attach to running application
and find you processes there.
I do some MPI-related debugging with log traces, but you can also run gdb if you're using mpich2: MPICH2 and gdb. This technique is a good practice in general when you're dealing with a process that's tricky to launch from a debugger.
You can use visual studio code which is free and much easier to work with than xterm. You duplicate the VS Code window and attach the debugger manually to each process. See the instruction in the below video: