Can't remove undesired trailing whitespace from variables used in slurm shell script that cause 'cd' to fail for too many arguments

Viewed 38

I am running code someone else developed to automate a process. It fails at the 'cd' step because a space is being inserted into the path its trying to generate. This is the whole shell script:

#!/bin/bash
#SBATCH --cpus-per-task=1
#SBATCH -n 1
#SBATCH -N 1
#SBATCH --mem=0 
#SBATCH --job-name=phase_2

t_nod=$2

file_path=`sed -n '1p' $3/$4/logs.txt`
protein=`sed -n '2p' $3/$4/logs.txt`

morgan_directory=`sed -n '4p' $3/$4/logs.txt`
smile_directory=`sed -n '5p' $3/$4/logs.txt`


python jobid_writer.py -pt $protein -fp $file_path -n_it $1 -jid $SLURM_JOB_NAME -jn $SLURM_JOB_NAME.txt

cd $file_path/$protein/iteration_$1
mkdir sdf
for f in smile/*
do
   tmp="$(cut -d'/' -f2 <<<"$f")"
   tmp="$(cut -d'_' -f1 <<<"$tmp")"
   if [ $tmp = train ];then name=training;fi
   if [ $tmp = valid ];then name=validation;fi
   if [ $tmp = test ];then name=testing;fi
   echo "#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1

echo \$1
echo \$2
echo \$3
oeomega classic -in \$1 -out sdf/\$2\_sdf.sdf -maxconfs 1 -strictstereo false -mpi_np \$3 -log \$2.log -prefix \$2">>$name'_'conf.sh

   sbatch -J $SLURM_JOB_NAME -c $t_nod $name'_'conf.sh $f $name $t_nod
done
wait

scancel $SLURM_JOBID

The script reads two variables from a log file:

file_path=`sed -n '1p' $3/$4/logs.txt`

protein=`sed -n '2p' $3/$4/logs.txt`

later the script wants to cd to that directory like so:

cd $file_path/$protein/

However, the script fails because for some reason, a trailing whitespace is added for the $variables so cd fails for too many arguments.

If I insert this into the script:

echo cd $file_path$protein

I get:

cd file_path protein

But I would expect to get

cd file_pathprotein

Things I've tried

  1. checking that log file for any trailing whitespaces (there are none)

  2. i tried adding quotes to the variables like

echo "$file_path$protein"

but that makes it even worse, now there are tab spaces between the variables when it is echo'd

  1. i tried adding quotes around the sed line, like so

file_path='sed -n '1p' $3/$4/logs.txt'

but then this caused two issues, one is that the variable became

$3/$4/logs.txt

when the script called it in the python part later AND it still had extra spaces in the cd command.

I consulted these two SO articles:

Bash: use cd command on a variable including spaces

https://askubuntu.com/questions/1361887/how-to-escape-whitespace-in-variable-passed-to-cd

And couldn't get these solutions to work. Partly because they didn't match my problem exactly and partly because I am very confused as to what is going on.

0 Answers
Related