BASH: How to write values generated by a for loop to a file quickly

user_10747564 0
Viewed 474

I have a for loop in bash that writes values to a file. However, because there are a lot of values, the process takes a long time, which I think can be saved by improving the code.

nk=1152
nb=24

for k in $(seq 0 $((nk-1))); do
    for i in $(seq 0 $((nb-1))); do
        for j in $(seq 0 $((nb-1))); do
            echo -e "$k\t$i\t$j" 
        done
    done
done > file.dat

I've moved the output action to after the entire loop is done rather than echo -e "$k\t$i\t$j" >> file.dat to avoid opening and closing the file many times. However, the speed the script writes to the file is still rather slow, ~ 10kbps.

Is there a better way to improve the IO?

Many thanks Jacek

io bash
5 Answers

It looks like the seq calls are fairly punishing since that is a separate process. Try this just using shell math instead:

for ((k=0;k<=$nk-1;k++)); do
    for ((i=0;i<=$nb-1;i++)); do
        for ((j=0;j<=$nb-1;j++)); do
            echo -e "$k\t$i\t$j" 
        done
    done
done > file.dat

It takes just 7.5s on my machine.

Another way is to compute the sequences just once and use them repeatedly, saving a lot of shell calls:

    nk=1152
    nb=24

    kseq=$(seq 0 $((nk-1)))
    bseq=$(seq 0 $((nb-1)))

    for k in $kseq; do
        for i in $bseq; do
            for j in $bseq; do
                echo -e "$k\t$i\t$j" 
            done
        done
    done > file.dat

This is not really "better" than the first option, but it shows how much of the time is spent spinning up instances of seq versus actually getting stuff done.

Bash isn't always the best for this. Consider this Ruby equivalent which runs in 0.5s:

#!/usr/bin/env ruby

nk=1152
nb=24

nk.times do |k|
  nb.times do |i|
    nb.times do |j|
      puts "%d\t%d\t%d" % [ k, i, j ]
    end
  end
end
user_871890

What is the most time consuming is calling seq in a nested loop. Keep in mind that each time you call seq it loads command from disk, fork a process to run it, capture the output, and store the whole output sequence into memory.

Instead of calling seq you could use an arithmetic loop:

#!/usr/bin/env bash

declare -i nk=1152
declare -i nb=24
declare -i i j k

for ((k=0; k<nk; k++)); do
  for (( i=0; i<nb; i++)); do
    for (( j=0; j<nb; j++)); do
      printf '%d\t%d\t%d\n' "$k" "$i" "$j"
    done
  done
done > file.dat
user_79398710

Running seq in a subshell consumes most of the time.

Switch to a different language that provides all the needed features without shelling out. For example, in Perl:

#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };

my $nk = 1152;
my $nb = 24;

for my $k (0 .. $nk - 1) {
    for my $i (0 .. $nb - 1) {
        for my $j (0 .. $nb - 1) {
            say "$k\t$i\t$j" 
        }
    }
}

The original bash solution runs for 22 seconds, the Perl one finishes in 0.1 seconds. The output is identical.

user_10306750

@Jacek : I don't think the I/O is the problem, but the number of child processes spawned. I would store the result of the seq 0 $((nb-1)) into an array and loop over the array, i.e.

nb_seq=( $(seq 0 $((nb-1)) )
...
  for i in "${nb_seq[@]}"; do
    for j in "${nb_seq[@]}"; do
user_19344280

seq is bad) once i've done this function special for this case:

$ que () { printf -v _N %$1s; _N=(${_N// / 1}); printf "${!_N[*]}"; }
$ que 10
0 1 2 3 4 5 6 7 8 9

And you can try to write first all to a var and then whole var into a file:

store+="$k\t$i\t$j\n"
printf "$store" > file

No. it's even worse like that)

user_126074430
Related