I have 10 cpu nodes and each node has 72 cpu cores. I am running a code that reads huge files that need to be read in parallel to take less time. But the problem is that it goes to the memory limit. So, I guess I am only running all of them on different cpu cores instead of different cpu nodes and am using only 1 node! The part of the code is as followed:
import multiprocessing
from joblib import Parallel, delayed
def Reading(ROI, j):
ds = datasets[j]
f = h5py.File(os.path.join(path_raw, ds),'r') # Opening the data file
data = f[data_string][:, ROI, 0, 0]
f.close()
return data
num_cores = multiprocessing.cpu_count()
data_tmp = Parallel(n_jobs=num_cores)(delayed(Reading)(ROI, j) for j in range(13))
In addition, the whole code (from start to end) should be done 8 times for 8 different datasets which I parallelized the whole code separately by using argparse (GNU parallel) library as follows:
def argparser():
parser = argparse.ArgumentParser()
parser.add_argument('--mod_num', type=int, default=0)
return parser
And the 'mod_num' parameter will be controlled by the following bash file:
#!/bin/bash
#SBATCH --nodes=10
#SBATCH --time=1-00:00:00
unset LD_PRELOAD
source /etc/profile.d/modules.sh
module purge
module load anaconda-python
parallel python myexample.py --mod_num ::: {0..7}
So, it means that I have two different kinds of parallel processing (one by joblib and one by GNU parallel), but both of them are using only 1 cpu node and I cannot use the other 9 cpu nodes!
It would be so appreciated if someone can help me to solve this.