while loops in parallel with input from splited file

You are looking for --pipe. In this case you can even use the optimized --pipepart (version >20160621):

export TAXONOMY=path/taxonomy.file.tsv
doit() {
while read line
do
#find hits
        hits=$(grep $line alnout_file | cut -f 2)
        completename=$(grep $line $TAXONOMY | sed 's/D_0.*D_4/D_4/g')
        printf "\nRate of taxonomies for this sequence in %%:\t$completename\n"
        printf "Taxonomy\t%aligned\tnumber_ocurrences_in_the_alignment\tnum_ocurrences_in_databank\t%alingment/databank\n"

        #find hits and calculate the frequence (%) of the taxonomy in the alignment output
        # ex.: Bacillus_subtilis 33
        freqHits=$(grep "${hits[@]}" $TAXONOMY | \
                cut -f 2 | \
                awk '{a[$0]++} END {for (i in a) {print i, "\t", a[i]/NR*100, "\t", a[i]}}' | \
                sed -e 's/D_0.*D_5/D_5/g' -e 's#\s\t\s#\t#g' | \
                sort -k2 -hr)

        # print frequence of each taxonomy in the databank

        freqBank=$(while read line; do grep -c "$line" $TAXONOMY; done < <(echo "$freqHits" | cut -f 1))
        #print cols with taxonomy and calculations
        paste <(printf %s "$freqHits") <(printf %s "$freqBank") | awk '{print $1,"\t",$2"%","\t",$3,"\t",$4,"\t",$3/$4*100"%"}'

done
}
export -f doit
parallel -a input_file --pipepart doit

This will chop input_file into 10*ncpu blocks (where ncpu is the number of CPU threads), pass each block to doit, run ncpu jobs in parallel.

That said I think your real problem is spawning too many programs: If you rewrite doit in Perl or Python I will expect you will see a major speedup.

As an alternative I threw together a quick test.

#! /bin/env bash
mkfifo PIPELINE             # create a single queue
cat "$1" > PIPELINE &       # supply it with records
{ declare -i cnt=0 max=14
  while (( ++cnt <= max ))  # spawn loop creates worker jobs
  do printf -v fn "%02d" $cnt
     while read -r line     # each work loop reads common stdin...
     do echo "$fn:[$line]"
        sleep 1
     done >$fn.log 2>&1 &   # these run in background in parallel
  done                      # this one exits
} < PIPELINE                # *all* read from the same queue
wait
cat [0-9][0-9].log

Doesn't need split, but does need a mkfifo.

Obviously, change the code inside the internal loop.

This answers what you asked, namely how to process in parallel the 14 files you get from running split. However, I don't think it is the best way of doing whatever it is that you are trying to do - but we would need some answers from you for that.

So, let's make a million line file and split it into 14 parts:

seq 1000000 > 1M
split -n 14 1M part-

That gives me 14 files called part-aa through part-an. Now your question is how to process those 14 parts in parallel - (read the last line first):

#!/bin/bash

# This function will be called for each of the 14 files
DoOne(){
   # Pick up parameters
   job=$1
   file=$2
   # Count lines in specified file
   lines=$(wc -l < "$file")
   echo "Job No: $job, file: $file, lines: $lines"
}

# Make the function above known to processes spawned by GNU Parallel
export -f DoOne

# Run 14 parallel instances of "DoOne" passing job number and filename to each
parallel -k -j 14 DoOne {#} {} ::: part-??

Sample Output

Job No: 1, file: part-aa, lines:    83861
Job No: 2, file: part-ab, lines:    72600
Job No: 3, file: part-ac, lines:    70295
Job No: 4, file: part-ad, lines:    70295
Job No: 5, file: part-ae, lines:    70294
Job No: 6, file: part-af, lines:    70295
Job No: 7, file: part-ag, lines:    70295
Job No: 8, file: part-ah, lines:    70294
Job No: 9, file: part-ai, lines:    70295
Job No: 10, file: part-aj, lines:    70295
Job No: 11, file: part-ak, lines:    70295
Job No: 12, file: part-al, lines:    70294
Job No: 13, file: part-am, lines:    70295
Job No: 14, file: part-an, lines:    70297

You would omit the -k argument to GNU Parallel normally - I only added it so the output comes in order.

I think that using a bunch of grep and awk commands is the wrong approach here - you would be miles better off using Perl, or awk. As you have not provided any sample files I generated some using this code:

#!/bin/bash

for a in {A..Z} {0..9} ; do
   for b in {A..Z} {0..9} ; do
      for c in {A..Z} {0..9} ; do
         echo "${a}${b}${c}"
      done
   done
done > a

# Now make file "b" which has the same stuff but shuffled into a different order
gshuf < a > b

Note that there are 26 letters in the alphabet, so if I add the digits 0..9 to the letters of the alphabet, I get 36 alphanumeric digits and if I nest 3 loops of that I get 36^3 or 46,656 lines which matches your file sizes roughly. File a now looks like this:

AAA
AAB
AAC
AAD
AAE
AAF

File b looks like this:

UKM
L50
AOC
79U
K6S
6PO
12I
XEV
WJN

Now I want to loop through a finding the corresponding line in b. First, I use your approach:

time while read thing ; do grep $thing b > /dev/null ; done < a

That takes 9 mins 35 seconds.

If I now exit grep on the first match, on average I will find it in the middle, which means the time will be halved since I won't continue to needlessly read b after I find what I want.

time while read thing ; do grep -m1 $thing b > /dev/null ; done < a

That improves the time down to 4 mins 30 seconds.

If I now use awk to read the contents of b into an associative array (a.k.a. hash) and then read the elements of a and find them in b like this:

time awk 'FNR==NR{a[$1]=$1; next} {print a[$1]}' b a > /dev/null

That now runs in 0.07 seconds. Hopefully you get the idea of what I am driving at. I expect Perl would do this in the same time and also provide more expressive facilities for the maths in the middle of your loop too.

I hope this small script helps you out:

function process {
    while read line; do
        echo "$line"
    done < $1
}

function loop {
    file=$1
    chunks=$2
    dir=`mktemp -d`
    cd $dir
    split -n l/$chunks $file
    for i in *; do
        process "$i" &
    done
    rm -rf $dir
}

loop /tmp/foo 14

It runs the process loop on the specified file with the specified number of chunks (without splitting lines) in parallel (using & to put each invocation in the background). I hope it gets you started.

This can do the job for You, I am not familiar with parallel instead using native bash spawning processes &:

function loop () {
  while IFS= read -r -d $'\n'
  do
    # YOUR BIG STUFF
  done < "${1}"
}

arr_files=(./xa*)
for i in "${arr_files[@]}"
do loop "${i}" &
done
wait