Here is my job script content:
#!/bin/bash
#SBATCH --partition=compute
#SBATCH --job-name=demo
#SBATCH --output=job.%j.out
#SBATCH --error=job.%j.err
#SBATCH -N 3
#SBATCH --ntasks-per-node=1
#SBATCH --export=ALL
srun --mpi=pmi2 -n 3 hostname
When I submit this job via sbatch, it runs to completion and return hostname of my nodes SUCCESSFULLY. But if I submit via Slurm REST API (slurm/v0.0.37/job/submit), the API returns HTTP 200 with job id, and the job result is FAILED with following stderr.
srun: error: auth_g_unpack: remote plugin_id 101 not found
srun: error: slurm_receive_msgs: [[node1]:6818] auth_g_unpack: Resource temporarily unavailable
srun: error: slurm_receive_msgs: [[node1]:6818] failed: Header lengths are longer than data received
srun: error: auth_g_unpack: remote plugin_id 101 not found
srun: error: slurm_receive_msgs: [[node3]:6818] auth_g_unpack: Resource temporarily unavailable
srun: error: slurm_receive_msgs: [[node3]:6818] failed: Header lengths are longer than data received
srun: error: auth_g_unpack: remote plugin_id 101 not found
srun: error: slurm_receive_msgs: [[node2]:6818] auth_g_unpack: Resource temporarily unavailable
srun: error: slurm_receive_msgs: [[node2]:6818] failed: Header lengths are longer than data received
The body I POST likes below:
{
"job": {
"name":"demo",
"partition":"compute",
"standard_output":"%j.out",
"standard_error":"%j.err",
"nodes":3,
"tasks":3,
"tasks_per_node":1,
"get_user_environment":1,
"current_working_directory":"/SHARED_NFS_STORAGE"
},
"script":"#!/bin/bash\nsrun --mpi=pmi2 -n 3 hostname"
}