Submit OpenMPI job via Slurm REST API failed

Viewed 195

Here is my job script content:

#!/bin/bash
#SBATCH --partition=compute
#SBATCH --job-name=demo
#SBATCH --output=job.%j.out
#SBATCH --error=job.%j.err
#SBATCH -N 3
#SBATCH --ntasks-per-node=1
#SBATCH --export=ALL
srun --mpi=pmi2 -n 3 hostname

When I submit this job via sbatch, it runs to completion and return hostname of my nodes SUCCESSFULLY. But if I submit via Slurm REST API (slurm/v0.0.37/job/submit), the API returns HTTP 200 with job id, and the job result is FAILED with following stderr.

srun: error: auth_g_unpack: remote plugin_id 101 not found
srun: error: slurm_receive_msgs: [[node1]:6818] auth_g_unpack: Resource temporarily unavailable
srun: error: slurm_receive_msgs: [[node1]:6818] failed: Header lengths are longer than data received
srun: error: auth_g_unpack: remote plugin_id 101 not found
srun: error: slurm_receive_msgs: [[node3]:6818] auth_g_unpack: Resource temporarily unavailable
srun: error: slurm_receive_msgs: [[node3]:6818] failed: Header lengths are longer than data received
srun: error: auth_g_unpack: remote plugin_id 101 not found
srun: error: slurm_receive_msgs: [[node2]:6818] auth_g_unpack: Resource temporarily unavailable
srun: error: slurm_receive_msgs: [[node2]:6818] failed: Header lengths are longer than data received

The body I POST likes below:

{
    "job":  {
    "name":"demo",
    "partition":"compute",
    "standard_output":"%j.out",
    "standard_error":"%j.err",
    "nodes":3,
    "tasks":3,
    "tasks_per_node":1,
    "get_user_environment":1,
    "current_working_directory":"/SHARED_NFS_STORAGE"
    },
    "script":"#!/bin/bash\nsrun --mpi=pmi2 -n 3 hostname"
}
0 Answers
Related