Alternative to using sed inside loop

Question

I want to replace multiple (non-consecutive) lines of a big file, using files that contain one single line. What I found to work is

for i in ${list[@]}; do
   line=$(cat file_$i.txt);
   sed -i "$i c $line" bigfile.txt;
done

Here, list contains the lines I want to replace. Here is an example:

$ list=(1 3 4 7)
$ cat file_1.txt
this is the new line 1
$ cat file_3.txt
this is the new line 3
$ cat file_4.txt
this is the new line 4
$ cat file_7.txt
this is the new line 7

$ cat bigfile.txt
line 1
line 2
line 3
line 4
line 5
line 6
line 7
line 8

The output of the above script is

$ cat bigfile.txt 
newline 1
line 2
newline 3
newline 4
line 5
line 6
newline 7
line 8

It works, but in every step of the loop sed opens and reads the whole file, as I understand it, so this method is very slow. What are the other faster ways to do this, preferably using sed?

user_5825294 · Answer

A possible solution:

sed "$(for i in $list; do echo "$i c $(cat file_$i.txt)"; done)" bigfile.txt

($list could have to be ${list[@]} or ${list[*]} or anything else, depending how it is constructed.)

Your original loop is used to build a Sed script each line of which is like 1 c content_of_file_1_dot_txt; this script is then run only once on bigfile.txt.

user_1745001 · Answer

Forget about the list[] array and run this in a directory containing just your files:

awk '
sub(/^file_/,"",FILENAME) { map[FILENAME+0] = $0; next }
{ print (FNR in map ? map[FNR] : $0) }
' file_*.txt bigfile.txt

or if you're set on using a list[] array too:

awk -v list="${list[*]}" '
BEGIN {
    split(list,tmp)
    for (i in tmp) {
        lineNrs[tmp[i]]
    }
}
NR in lineNrs {
    if ( (getline line < ("file_" NR ".txt")) > 0 ) {
        $0 = line
    }
    close("file_" NR ".txt")
}
{ print }
' bigfile.txt

user_967492 · Answer

This might work for you (GNU sed and parallel):

<<<"${list[@]}" sed -E 's/\S+/&r file_&.txt
&d
/g' | sed -i -f - bigfile

Or using GNU parallel:

parallel 'echo "{}r file_{}.txt";echo "{}d"' ::: ${list[@]} | sed -i -f - bigfile

Or if the replacement text files only contain one line each:

parallel echo '{}c$(



Or you might prefer to use the file names in the current directory:

parallel --rpl '{@} s/[^0-9]//g' 'echo "{@}r {}";echo "{@}d"' ::: file_* |
sed -i -f - bigfile

Alternative to using sed inside loop

3 Answers