Bash subshell to file

user_5783481 0
Viewed 144

I'm looping over a large file, on each line I'm running some commands, when they finish I want the entire output to be appended to a file.

Since there's nothing stopping me from running multiple commands at once, I tried to run this in the background &.

It doesn't work as expected, it just appends the commands to the file as they finish, but not in the order they appear in the subshell

#!/bin/bash
while read -r line; do
  (
    echo -e "$line\n-----------------"
    trivy image --severity CRITICAL $line
    # or any other command that might take 1-2 seconds
    echo "============="
  ) >> vulnerabilities.txt &
done <images.txt

Where am I wrong?

subshell shell bash
1 Answers

Consider using GNU Parallel to get lots of things done in parallel. In your case:

parallel -k -a images.txt trivy image --severity CRITICAL > vulnerabilities.txt

The -k keeps the output in order. Add --bar or --eta for progress reports. Add --dry-run to see what it would do without actually doing anything. Add -j ... to control the number of parallel jobs at any one time - by default, it will run one job per CPU core at a time - so it will basically keep all your cores busy till the jobs are done.

If you want to do more processing on each line, you can declare a bash function and call that with each line as its parameter... see here.

user_28366210
Related