Segmentation fault with MPI shared memory and quad precision

user_2249356 0
Viewed 193

A very simple C program using MPI shared memory crashes for me when quadruple precision (__float128) is used with GCC (but not with the Intel C compiler). The program allocates a shared real array one element long for every MPI process (allocation is done by the master process and then accessed by other processes). Each process writes into one of the elements and prints another element to standard output. When using standard built-in types float and double (or just single process), there are no problems at all. When using __float128 (and more than one process), the highlighted line below results in segmentation fault for all processes.

#include <mpi.h>
#include <stdio.h>

const int verbose = 0;

/* typedef float real; */
/* typedef double real; */
typedef __float128 real;

#define _(FUN, ARGS) \
{ \
    int status = FUN ARGS; \
    if (verbose) printf(#FUN " exit code: %d\n", status); \
}

int main (int argc, char* argv[])
{
    int nprocs, rank, assert = 0, disp = 1, master = 0, bytesize = sizeof(real);
    void* baseptr = NULL;
    real* arr;
    MPI_Win win;

    printf("Data type byte size: %d\n", bytesize);

    _(MPI_Init,      (&argc, &argv))
    _(MPI_Comm_size, (MPI_COMM_WORLD, &nprocs))
    _(MPI_Comm_rank, (MPI_COMM_WORLD, &rank))

    MPI_Aint totsize = rank == 0 ? bytesize * nprocs : 0;

    _(MPI_Win_allocate_shared, (totsize, disp, MPI_INFO_NULL, MPI_COMM_WORLD, &baseptr, &win))
    _(MPI_Win_shared_query,    (win, master, &totsize, &disp, &arr))
    _(MPI_Win_fence,           (assert, win))

    arr[rank] = rank + 1;  /*  <-- Segmentation fault here when using __float128 */

    _(MPI_Win_fence, (assert, win))

    printf("Element %d: %g\n", (rank + 1) % nprocs, (double)arr[(rank + 1) % nprocs]);

    _(MPI_Win_fence, (assert, win))
    _(MPI_Win_free,  (&win));
    _(MPI_Finalize,  ());

    return 0;
}

The program works in some configurations, but not in others. Here are those that I was able to check:

Compiler Open MPI 4.1.0 Intel MPI (oneAPI 2021)
Intel oneAPI 2021 mpiicc OK for all types OK for all types
GCC 9.3 mpicc FAILS with __float128 N/A
GCC 10.2 mpicc FAILS with __float128 N/A

I use openSUSE Tumbleweed 20210319.

On the first sight, this seems to be a problem in GCC. Or am I missing something in the code?

shared-memory mpi intel gcc c
1 Answers

What happens here is MPI_Win_allocate_shared() returns a memory area aligned on 8 bytes, but GCC assumes arr is aligned on 16 bytes and a crash can occur if arr is not aligned on 16 bytes.

You can work around this issue by manually realigning the data.

#define REALIGN(a, type) \
    ((a) + sizeof(type) - 1 & ~(sizeof(type) - 1))

and then

    MPI_Aint totsize = rank == 0 ? (bytesize * (nprocs+1)) : 0;

and finally

    arr = (real *)REALIGN((unsigned long)arr, real);

FWIW, on recent Intel processors, aligned and non aligned instructions run at the same speed when the data is aligned, and that is likely why the Intel compiler do not generate aligned instructions any more.

user_80624910
Related