SLURM Batch file still needs #SBATCH calls for resource allocation when using dask-jobqueue?

Viewed 59

Until now I have been running a code on a HPC with SLURM using only one node, parallelizing the computation using joblib. In order to submit the jobs I had a job submission file as


#SBATCH --time=60:00:00
#SBATCH --output=windows.out
#SBATCH --error=windows.err
#SBATCH --ntasks=40
#SBATCH --ntasks-per-node=40

# Load environment
module load conda/py3-latest
source activate myenvpy3

cd /mainfs/home/PEDRO/

# Run Python parallel program
python individual_validate_windows.py 40

exit 

I am trying to use dask.distributed and dask-jobqueue to use more resources available at the HPC. My main doubt is whether or not I still need the #SBATCH calls to allocate resources in the batch file or if this is already handled using cluster and scale from dask-jobqueue inside my Python code. Also, would I still need to specify the n_jobs parameter for joblib.parallel when using dask as backend?

    cluster = SLURMCluster(cores=40,
                           memory='160 GB',
                           walltime='60:00:00',
                           job_extra=['--output=individual_validate_windows.out',
                                      '--error=individual_validate_windows.err',
                                      '--mail-type=ALL'])

    cluster.scale(3)
    client = Client(cluster)

    # Executing analysis in parallel
        with joblib.parallel_backend('dask'):
            parallel = Parallel(verbose=1, n_jobs=-2)
            out = parallel(delayed(cv_and_fit)(features, target, params, train_index, test_index)
                           for params, (train_index, test_index) in product(param_grid, kf.split(features)))


Any help will be appreciated.

0 Answers
Related