delete entries at certain indices in space delimited text file

user_12334018 0
Viewed 59

I have a .txt file with numeric indices of certain 'outlier' data points, each on their own line, called by $outlier_file:

1 
7 
30 
43 
48 
49 
56 
57 
65

Using the following code, I can successfully remove certain files (volumes of neuroimaging data in this case) by using while + read.

while read outlier; do
    # Remove current outlier vol from eddy unwarped DWI data
    rm $DWI_path/$1/vol000*"$outlier".nii.gz;
done < $outlier_file

However, I also need to remove the numbers located at these 'outlier' indices from another text file stored in $bvec_file, which has 69 columns & 3 rows. Within each row, the numbers are space delimited. So e.g., for this example, I need to remove all 3 rows of column 1, 7, 30, etc. and then save this version with the outliers removed into a new *.txt file.

0  0.9988864166  -0.0415925034  -0.06652866169  -0.6187155495  0.2291534462  0.8892356214  0.7797364286  0.1957395685  0.9236669465  -0.5400265342  -0.3845263463  -0.4903989539  0.4863306385  -0.6496130843  0.5571164636  0.8110081715  0.9032142094  -0.3234596075  -0.1551409525  -0.806059879  0.4811597826  -0.7820757748  -0.9528881463  0.1916556621  -0.007136403284  -0.2459431735  -0.7915263574  -0.1938049261  -0.1578786349  0.8688043633  -0.5546072294  -0.4019951732  0.2806154851  0.3478762022  0.9548067252  -0.9696777541  -0.4816255837  -0.7962240023  0.6818610905  0.7097978218  0.6739686799  0.1317547111  -0.7648252249  -0.1456021218  -0.5948047487  0.0934205064  0.5268769564  -0.8618324858  -0.3721029232  -0.1827616535  0.691353613  0.4159071597  0.4605505287  0.1312199424  0.426674893  -0.4068291509  0.7167859082  0.2330824665  0.01909161256  -0.06375254731  -0.5981122948  -0.2672253674  0.6875472994  0.2302943724  0  0  0  0  
0  0.04258194557  0.9988207007  0.6287131425  0.7469024143  0.5528476637  0.3024964957  0.1446931241  0.9305823612  0.1675139932  0.8208211337  0.8238722992  0.5983722761  0.4238174961  0.639429196  0.1072148887  0.5551578885  0.003337599176  0.511740508  0.9516619405  0.3851404227  0.8526321065  0.1390947346  0.2030449535  0.7759459569  0.165587903  0.9523372297  0.5801228933  0.3277276562  0.7413928896  0.442482978  0.2320585706  0.1079269171  0.1868672655  0.1606136006  0.2968573235  0.1682337977  0.8745679247  0.5989061899  0.4172933119  0.01746934331  0.5641480832  0.7455469091  0.3471016571  0.8035001467  0.5870623128  0.361107261  0.8192579877  0.4160218909  0.5651330299  0.4070513153  0.7221181184  0.714223583  0.6971767133  0.4937978446  0.4232911691  0.8011701162  0.2870385494  0.9016941521  0.09688949547  0.9086826131  0.2631932421  0.152678096  0.6295753848  0.9712458578  0  0  0  0  
0  -0.02031513434  -0.02504539005  -0.7747862425  0.2435730944  0.8011542666  0.343155766  -0.6091592581  -0.3093581909  -0.3446424728  -0.1860752773  -0.4163819443  -0.6336083058  0.7641081337  -0.4112580017  -0.8234841915  0.1845683194  0.4291770641  -0.7959243273  -0.2650864686  0.449371034  -0.203724703  0.6074620459  0.2253373638  -0.6009791836  -0.9861692137  0.1804598471  0.1922068008  -0.9246806119  0.6522353256  -0.2222336438  0.7990992685  -0.9092588527  -0.9414539684  0.9236803664  0.0148272357  -0.1772637652  0.05628269894  -0.08566629406  -0.6007759525  0.7041888058  0.4769729119  0.6532997034  -0.5427364139  -0.5772239915  0.5491494803  0.9278330427  0.2263117816  -0.290121617  0.7363179158  0.8949343019  -0.02399176716  0.5629439653  -0.5493977074  -0.8596191107  -0.7992328333  0.4388809483  0.6354737076  0.3641705918  0.9951120218  0.412591228  -0.75696169  0.9514620339  -0.3618197699  0.06038199928  0  0  0  0

As far as I've gotten in one approach is using awk to index the right columns.. (just printing them right now) but I can only get this to work if I call $1 (i.e., the numeric index of the first outlier column)...

awk -F ' ' '{print $1}' $bvec_file

If I try to refer to the value in $outlier, it doesn't work. Instead, this prints the entire contents of $bvec_file

while read outlier; do
    
    # Remove current outlier vol from eddy unwarped DWI data
    rm $DWI_path/$1/vol000*"$outlier".nii.gz;
    
    # Remove outlier #'s from bvec file 
    awk -F ' ' '{print $1}' $bvec_file

done < $outlier_file

I am completely stuck on how to get this done. Any advice would be greatly appreciated.

sh while-loop awk shell bash
1 Answers

To delete the outliers from bvec_file after the loop and only delete the ones where the associated file was successfully removed:

#!/usr/bin/env bash

tmp=$(mktemp) || exit 1
while IFS= read -r outlier; do
    
    # Remove current outlier vol from eddy unwarped DWI data
    rm "$DWI_path/$1"/vol000*"$outlier".nii.gz &&
    echo "$outlier"

done < "$outlier_file" |
awk '
    NR==FNR { os[$0]; next }
    {
        for (o in os) {
            $o=""
        }
        $0=$0; $1=$1
    }
1' - "$bvec_file" > "$tmp" &&
mv "$tmp" "$bvec_file"

Or to delete the outliers one at a time as the files are removed:

#!/usr/bin/env bash

tmp=$(mktemp) || exit 1
while IFS= read -r outlier; do
    
    # Remove current outlier vol from eddy unwarped DWI data
    rm "$DWI_path/$1"/vol000*"$outlier".nii.gz &&
    
    # Remove outlier #'s from bvec file 
    awk -v o="$outlier" '{$o=""; $0=$0; $1=$1} 1' "$bvec_file" > "$tmp" &&
    mv "$tmp" "$bvec_file"

done < <(sort -rnu "$outlier_file")

Always quote your shell variables, see https://mywiki.wooledge.org/Quotes, and the && at the end of each line is to ensure the next command only runs if the previous commands succeeded.

The magical incantation in the awk script does the following - lets say your input is a b c and the outlier field is field number 2, b:

$ echo 'a b c'
a b c
$
$ echo 'a b c' | awk -v o=2 '{$o=""; print NF ":", $0}'
3: a  c
$
$ echo 'a b c' | awk -v o=2 '{$o=""; $0=$0; print NF ":", $0}'
2: a  c
$
$ echo 'a b c' | awk -v o=2 '{$o=""; $0=$0; $1=$1; print NF ":", $0}'
2: a c

The o="" sets the field value to null, the $0=$0 forces awk to resplit $0 into fields so it effectively deletes field 2 (as opposed to the previous step which set it to null but it still existed as such), and the $1=$1 recombines $0 from it's fields replacing every FS (any contiguous chain of white space chars including the 2 blanks now between a and c) with OFS (a single blank char).

user_17450010
Related