Until now I have been running a code on a HPC with SLURM using only one node, parallelizing the computation using joblib. In order to submit the jobs I had a job submission file as
#SBATCH --time=60:00:00
#SBATCH --output=windows.out
#SBATCH --error=windows.err
#SBATCH --ntasks=40
#SBATCH --ntasks-per-node=40
# Load environment
module load conda/py3-latest
source activate myenvpy3
cd /mainfs/home/PEDRO/
# Run Python parallel program
python individual_validate_windows.py 40
exit
I am trying to use dask.distributed and dask-jobqueue to use more resources available at the HPC. My main doubt is whether or not I still need the #SBATCH calls to allocate resources in the batch file or if this is already handled using cluster and scale from dask-jobqueue inside my Python code. Also, would I still need to specify the n_jobs parameter for joblib.parallel when using dask as backend?
cluster = SLURMCluster(cores=40,
memory='160 GB',
walltime='60:00:00',
job_extra=['--output=individual_validate_windows.out',
'--error=individual_validate_windows.err',
'--mail-type=ALL'])
cluster.scale(3)
client = Client(cluster)
# Executing analysis in parallel
with joblib.parallel_backend('dask'):
parallel = Parallel(verbose=1, n_jobs=-2)
out = parallel(delayed(cv_and_fit)(features, target, params, train_index, test_index)
for params, (train_index, test_index) in product(param_grid, kf.split(features)))
Any help will be appreciated.