How to concatenate gathered data using mpi4py library in python

Question

I used to list append of data employing mpi4py and try to save the data sequentially at the source(root==0) node.

As suggested by Alan22, I've modified the code and it works, but the script does not concatenate properly, so I get the output file as shown in attached figure:01.

Can anybody help how to fix the error message? In addition, whatever I've written in python script [shown below], isn't the best way to solve the problem.

Is there any way to solve this type of problem efficiently? Any help is highly appreciated.

The python script is given as follows:

import numpy as np
from scipy import signal
from mpi4py import MPI  
import random
import cmath, math
import matplotlib.pyplot as plt
import time

#File storing path
save_results_to = 'File storing path'

count_day = 1
count_hour = 1

arr_x = [0, 8.49, 0.0, -8.49, -12.0, -8.49, -0.0, 8.49, 12.0]
arr_y = [0, 8.49, 12.0, 8.49, 0.0, -8.49, -12.0, -8.49, -0.0]
M = len(arr_x)
N = len(arr_y)

np.random.seed(12345)
total_rows = 50000
raw_data=np.reshape(np.random.rand(total_rows*N),(total_rows,N))

# Function of CSD:: Using For Loop
fs = 500;       # Sampling frequency
def csdMat(data):
    dat, cols = data.shape   # For 2D data
    total_csd = []
    for i in range(cols):
        col_csd =[]
        for j in range( cols):
            freq, Pxy = signal.csd(data[:,i], data[:, j], fs=fs, window='hann', nperseg=100, noverlap=70, nfft=5000) 
            col_csd.append(Pxy)  
        total_csd.append(col_csd)
        pxy = np.array(total_csd)
    return freq, pxy

# Finding cross spectral density (CSD)
t0 = time.time()
freq, csd = csdMat(raw_data)
print('The shape of the csd data', csd.shape)
print('Time required {} seconds to execute CSD--For loop'.format(time.time()-t0))

kf=1*2*np.pi/10
resolution = 50 # This is important:: the HIGHER the Resolution, the higher the execution time!!!
grid_size = N * resolution
kx = np.linspace(-kf, kf, )  # space vector
ky = np.linspace(-kf, kf, grid_size)  # space vector

def DFT2D(data):
    P=len(kx)
    Q=len(ky)
    dft2d = np.zeros((P,Q), dtype=complex)
    for k in range(P):
        for l in range(Q):
            sum_log = []
            mat2d = np.zeros((M,N))
            sum_matrix = 0.0
            for m in range(M):
                for n in range(N):
                    e = cmath.exp(-1j*((((dx[m]-dx[n])*kx[l])/1) + (((dy[m]-dy[n])*ky[k])/1)))
                    sum_matrix += data[m, n] * e
            dft2d[k,l] = sum_matrix
    return dft2d

dx = arr_x[:]; dy = arr_y[:]


comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()

data = []
start_freq = 100
end_freq   = 109
freq_range = np.arange(start_freq,end_freq)
no_of_freq = len(freq_range)

for fr_count in range(start_freq, end_freq):
    if fr_count % size == rank:
        dft = np.zeros((grid_size, grid_size))
        spec_csd = csd[:,:, fr_count]
        dft = DFT2D(spec_csd)  # Call the DFT2D function
        spec = np.array(np.real(dft))  # Spectrum or 2D_DFT of data[real part]
        print('Shape of spec', spec.shape)
        data.append(spec)
        #data = np.append(data,spec)
        np.seterr(invalid='ignore')
data = comm.gather(data, root =0)
#    comm.Allreduce(MPI.IN_PLACE,data,op=MPI.MAX)
print("Rank: ", rank, ". Spectrum shape is:
", spec.shape)


if rank == 0:
    output_data = np.concatenate(data, axis = 0)
    #output_data = np.c_(data, axis = 0)
    dft_tot = np.array((output_data), dtype='object')
    res = np.zeros((grid_size, grid_size))
    for k in range(size):
        for i in range(no_of_freq):

            jj = np.around(freq[freq_range[i]], decimals = 2)

            #print('The shape of data after indexing', data1.shape)
            #data_final=data1.reshape(data1.shape[0]*data1.shape[1], data1.shape[2])
            res[i * size + k] = dft_tot[k][i] #np.array(data[k])
            data = np.array(res)
            #print('The shape of the dft at root node', data.shape)
            np.savetxt(save_results_to + f'Day_{count_day}_hour_{count_hour}_f_{jj}_hz.txt', data.view(float))

I use the following bash script command to run the script ( i.e., my_file.sh). I submit the job with command sbatch my_file.sh

#! /bin/bash -l
#SBATCH -J testmvapich2
#SBATCH -N 1 ## Maximum 04 nodes
#SBATCH --ntasks=10
#SBATCH --cpus-per-task=1        # cpu-cores per task
#SBATCH --mem-per-cpu=3000MB
#SBATCH --time=00:20:00
#SBATCH -p para
#SBATCH --output="stdout.txt"
#SBATCH --error="stderr.txt"
#SBATCH -A camk
##SBATCH --mail-type=ALL
##SBATCH --chdir=/work/cluster_computer/my_name/data_work/MMC331/


eval "$(conda shell.bash hook)"
conda activate myenv
#conda activate fast-mpi4py

cd $SLURM_SUBMIT_DIR

#module purge
#module add mpi/mvapich2-2.2-x86_64

mpirun python3 mpi_test.py

user_18183870 · Answer

You can try with this after "data = comm.gather(data, root=0)"

if rank == 0:
    print('Type of data:', type(data))
    dft_tot = np.array((data))#, dtype='object')
    print('shape of DATA array:', dft_tot.shape)
    #print('Type of dft array:', type(dft_tot))
    res = np.zeros((450,450))
    for k in range(size):
#            for i in range(len(data[rank])):
        for i in range(no_of_freq):

            jj = np.around(freq[freq_range[k]], decimals = 2)
            #data1 = np.array(dft_tot[k])
            res[i * size + k] = data[k]
            data = np.array(res)#.reshape(data1.shape[0]*data1.shape[1], data1.shape[2])
            print('The shape of the dft at root node', data.shape)
            np.savetxt(save_results_to + f'Day_{count_day}_hour_{co

Here is the link. Hope it helps mpi4py on HPC: comm.gather

user_4796894 · Answer

As mentioned in the comments, there are two typos in the code:

The indices for arrays kx and ky have been swapped in the line where variable e is calculated in the function DFT2D(data).
The code is being run for 10 MPI processes for frequencies fr_count in the range start_freq = 100 and end_freq = 109. For this, the loops and arange must be written as for fr_count in range(start_freq, end_freq + 1) and freq_range = np.arange(start_freq, end_freq + 1) as these are not end-point inclusive.

The data = comm.gather(data, root=0) and subsequent output_data = np.concatenate(data, axis=0) operations are performing as they should and as such, the question detracts from the actual issue in the code.

A major issue is that in line res[i * size + k] = dft_tot[k][i] arrays of disparate sizes are being assigned to each other.

Shape of res: 450 x 450
Shape of dft_tot: 10 x 50 x 450

The value of i*size + k ranges from 0 to 110. I think the user expects dft_tot to have the shape 450 x 450, probably due to the indexing confusion mentioned in typo#2 above. Properly done concatenation would yield dft_tot with shape 500 x 450 (since there are 10 arrays of size 50 x 450).

Currently the gather operation returns a list of lists, each containing a NumPy array of size 50 x 450. Technically, it should return a list of NumPy arrays each of size 50 x 450. Adding the line data = data[0] (since data has only one element anyway in each process) before performing data = comm.gather(data, root=0) will achieve this result.

But this whole process seems redundant..

Because there are 10 frequencies considered here. For each frequency, there is a data set of size 50 x 450 . There are 10 MPI processes with each handling one frequency out of the 10. Finally, 10 files are being written corresponding to each frequency. This makes the whole gather operation redundant, as each MPI process can directly write the file corresponding to each frequency.

If instead the dft_tot file was being written as is by rank = 0, then the gather operation would make sense. But splitting the array into the constituent frequencies defeats the point.

This achieves the same result without the gather operation:

comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()

start_freq = 100
end_freq   = 109
freq_range = np.arange(start_freq,end_freq+1)
no_of_freq = len(freq_range)

for fr_count in range(start_freq, end_freq+1):
    if fr_count % size == rank:
        dft = np.zeros((grid_size, grid_size))
        spec_csd = csd[:,:, fr_count]
        dft = DFT2D(spec_csd)  # Call the DFT2D function
        spec = np.array(np.real(dft))  # Spectrum or 2D_DFT of data[real part]
        print('Shape of spec', spec.shape)
        jj = np.around(freq[freq_range[rank]], decimals = 2)
        np.savetxt(f'Day_{count_day}_hour_{count_hour}_f_{jj}_hz.txt', spec.view(float))

How to concatenate gathered data using mpi4py library in python

2 Answers