Wait for subshell process to complete

Question

processUsageFile()
{
    #sdate=`pin_virtual_time  | awk -F" " '{print $3}'`;

    #Get all new files to be loaded to brm staging data.
    count=`ls ${PRE_STAGING}/TWN* 2>/dev/null|grep -v reprocess|wc -l`
    if [ $count -ne 0 ];then
        # Fork subshell
        (./efx_omc_brm_rpt_process.sh -t TWN & )&
        exitOnError
    fi

    #Process Rapid Report files
    count=`ls $PRE_STAGING/RR* 2>/dev/null|grep -v  reprocess|wc -l`
    if [ $count -ne 0 ];then
        (./efx_omc_brm_rpt_process.sh -t RR &)&
        exitOnError
    fi
...
...
}
#Reprocessing. Process the reprocessed files.
#This method updates the records in the BRM staging table.
reprocessingUsageFile()
{
    #Process TWN fulfillment reprocess files
    count=`ls $PRE_STAGING/TWN*reprocess* 2>/dev/null|wc -l`
    if [ $count -ne 0 ];then
        # Fork subshell
        (./efx_omc_brm_rpt_reprocess.sh -t TWN & ) &
    fi

    #Process Rapid Report files
    count=`ls $PRE_STAGING/RR*reprocess* 2>/dev/null|wc -l`
    if [ $count -ne 0 ];then
        (./efx_omc_brm_rpt_reprocess.sh -t RR &) &
    fi
...
...
}

#Pre processing
PreProcessing

# Start processing usage files.
processUsageFile

processErrFile

The idea of the above code is do parallel processing. All methods invoke multiple subshells and detach from tty. I would like to know if there is way to wait for first two methods to complete execution first and then run the last method.

Waiting for PIDs is somehow not accurate. Still trying...

waitPids() {
echo "Testing $pids -- ${#pids[@]}"
    while [ ${#pids[@]} -ne 0 ]; do
            local range=$(eval echo {0..$((${#pids[@]}-1))})
            local i
            for i in $range; do
                if ! kill -0 ${pids[$i]} 2> /dev/null; then
                    echo "Done -- ${pids[$i]}"
                     unset pids[$i]
                fi
            done
            pids=("${pids[@]}") 
            sleep 1
        done
    }

user_10787909 · Answer

The best way I found to parallelize and wait is to export a function to use in subshell and use xargs with -P for maximum number of parallel threads while feeding specific number of arguments to the work function with -n or -L.

from: https://man7.org/linux/man-pages/man1/xargs.1.html

       -P max-procs, --max-procs=max-procs
              Run up to max-procs processes at a time; the default is 1.
              If max-procs is 0, xargs will run as many processes as
              possible at a time.  Use the -n option or the -L option
              with -P;

Sample code:

# define some work function and export it
function unit_action() {
  echo action $*
  sleep 5
  echo action $* done
}
export -f unit_action

# list all arguments to feed into function
# with 2 parameters at a time in a maximum of 3 parallel threads
echo {1..9} | xargs -t -n 2 -P 3 bash -c 'unit_action $@' --
echo all done

xargs will implicitly wait until all input is consumed so no need for explicit wait command.

Wait for subshell process to complete

4 Answers