I'm trying to run an application in Singularity across nodes (864 MPI tasks) on an HPC system, namely the S4 machine at the University of Wisconsin's Space Science and Engineering Center (SSEC).
I'm using what Singularity describes as the hybrid model 1, meaning that I'm using the native (system) MPI but I also have MPI installed in the container. The mpi versions are compatible - I'm using Intel MPI version 17.0.6 outside the container and Intel MPI version 17.0.1 inside the container. The code in the container is compiled with Intel 17.0.1 compilers (C++, C, and Fortran).
So here's the problem. When I first ran the code, it complained about not finding the slurm configuration file:
fv3jedi_var.x: error: s_p_parse_file: unable to status file /etc/slurm-llnl/slurm.conf: No such file or directory, retrying in 1sec up to 60sec
So I found the system slurm.conf file in /etc/slurm and mounted this directory in the container as /etc/slurm-llnl. It now finds the configuration file but it does not understand the site-specific configuration:
fv3jedi_var.x: error: "ALL" is not a valid option for "EnforcePartLimits"
fv3jedi_var.x: error: Parsing error at unrecognized key: Features
fv3jedi_var.x: error: Parse error in file /etc/slurm-llnl/slurm.conf line 225: " Features=ivy"
fv3jedi_var.x: error: Parsing error at unrecognized key: Features
fv3jedi_var.x: error: Parse error in file /etc/slurm-llnl/slurm.conf line 226: " Features=ivy"
fv3jedi_var.x: error: Parsing error at unrecognized key: Features
[...]
So, I'm stuck. I'm guessing that this might be a PMI issue? I currently have slurm libpmi.so installed in the container and that's what I'm specifying with the I_MPI_PMI_LIBRARY variable. But I wonder if the native (system) PMI (I know it is PMI as opposed to PMI2 or PMIx) is somehow configured to properly process the system slurm.conf file? I have tried to use the native PMI library by mounting (binding) the appropriate directory in the container and changing my I_MPI_PMI_LIBRARY variable. But, the native PMI library is in the same directory as the glibc library and when I mount that there is a conflict between the glibc libraries inside and outside the container:
/bin/sh: relocation error: /usr/lib64/libc.so.6: symbol _dl_starting_up, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference
Any ideas on how to proceed? My slurm batch script is below. Thanks!
#!/usr/bin/bash
# --mem-per-cpu=8192M
#SBATCH --job-name=bm_con14
#SBATCH --partition=ivy
#SBATCH --ntasks=864
#SBATCH --cpus-per-task=1
#SBATCH --time=2:00:00
#SBATCH --mail-user=miesch@ucar.edu
source /etc/bashrc
module purge
module load license_intel
module load intel/17.0.6
ulimit -s unlimited
cd /data/users/mmiesch/runs/con-benchmark/con
JEDICON=/data/users/mmiesch
JEDIBUILD=/data/users/mmiesch/jedi/fv3-bundle/build-con
JEDIBIN=/data/users/mmiesch/jedi/fv3-bundle/build-con/bin
export SINGULARITY_BINDPATH="$JEDIBUILD,/etc/slurm:/etc/slurm-llnl"
srun --ntasks=864 --cpu_bind=cores --distribution=block:block --verbose singularity exec --home=$PWD $JEDICON/jedi-intel17-impi-hpc-dev.sif ${JEDIBIN
}/fv3jedi_var.x Config/3dvar_bump.yaml
exit 0