download two paired files in loop function before proceeding

user_2300940 0
Viewed 65

I have a loop that first downloads a file using a url link then processes it with samtools. For each file in input.txt there is one *.cram and one *.crai file which have the same name except the ending. Both these files need to be downloaded before samtools can be run. I wonder how to download both the *.cram and one *.crai using mapfile before proceeding to the samtools step.

test.txt

gs://hmf-cram-1231313-drup01010104/file1_dedup.realigned.cram 
gs://hmf-cram-1231313-drup01010104/file1_dedup.realigned.crai
gs://hmf-cram-1234235353-drup01242424/file1_dedup.realigned.cram 
gs://hmf-cram-1234235353-drup01242424/file1_dedup.realigned.crai

script

#!/bin/bash
mapfile -s 1 -t files < test.txt 
echo "${files[@]}"
for FILE in ${files[@]}; do
        gsutil -u absolute-bison-xxxx cp $FILE gs://bucket_1
        samtools view -@5  -O bam -f 4 ${FILE##*/}.cram > /home/user/${FILE##*/}.unmapped.bam
        rm *.cram
        rm *.crai;
done
loops bash
3 Answers

I would pass test.txt to the script when you invoke it:

./script < test.txt

Then process a record at a time. In this case it appears to be two lines:

while read cram && read crai
do
   process "$cram" "$crai"
done

I have not used mapfile, gsutil and samtools, btw, so I don't know who the two files come into play. However, I would validate the assumption that the two files are a matching pair and avoid the wildcard rm:

process() {
   local cram=$1
   local crai=$2

   if [ "$cram" != "${crai%crai}cram" ]
   then
       echo "Pair does not match: cram=$cram, crai=$crai" 1>&2
       return
    fi

    ...

    rm -- "${cram##*/}" "${crai##*/}"
}
Allan_Wind_97060

xargs may help you better here. First point is your input test.txt has odd lines with a trailing space, so I opted to deal with this:

xargs -l1 -a test.txt sh -c ';
basefile="${1##*/}";
basefile="${basefile%.*}";
if \
  gsutil -u absolute-bison-xxxx cp "$1" gs://bucket_1 \
  && samtools view -@5  -O bam -f 4 "$basefile.cram" \
  >"/home/user/$basefile.unmapped.bam";
then
  rm -- "./$basefile.cra"[im];
fi;
' _

Here the shell script run for each argument provided by xargs:

#!/usr/bin/env sh

# Remove leading path
basefile="${1##*/}"

# Remove last trailing dot extension
basefile="${basefile%.*}"

# Do the cloud stuffs
if \
  gsutil -u absolute-bison-xxxx cp "$1" gs://bucket_1 \
  && samtools view -@5  -O bam -f 4 "$basefile.cram" \
    >"/home/user/$basefile.unmapped.bam"
then
  # Cleanup .cran and .crai files
  rm -- "./$basefile.cra"[im]
fi
user_79398710

I wonder how to download both the *.cram and one *.crai using mapfile before proceeding to the samtools step.

Using mapfile but inside a while loop.

while mapfile -n2 -t cram_crai && ((${#cram_crai[@]} == 2)); do
  echo "${cram_crai[@]}"
  # The rest of your script here.
done < test.txt

Just in case you want to process more lines at a time just increase the value of 2's.

First element in the array is 0 which is *.cram

"${cram_crai[0]}"

Second element is 1 which is *.crai

"${cram_crai[1]}"

Parameter Expansion also applies with array expansion.

"${cram_crai[@]##*/}"
user_44522650
Related