Run multiple Google Apps Script clasp commands in parallel using a Bash script

I'd suggest the following solution:

#!/usr/bin/env bash

# use Node 14.17.5 to prevent "Error: Looks like you are offline." errors
# (see https://github.com/google/clasp/issues/872)
[ -s "/usr/local/opt/nvm/nvm.sh" ] && . "/usr/local/opt/nvm/nvm.sh"
nvm install 14.17.5
nvm use 14.17.5

# Check and process command line
if (( $# < 1 )); then
    echo "Usage: $(basename "$0") ACTION [ARG]..."
    exit 2
fi
action="$1"
args=("${@:2}")

# Define cleanup handler, create temporary log directory
trap '[[ -n "$(jobs -p)" ]] && kill -- -$$; [[ -n "${logdir}" ]] && rm -rf "${logdir}"' EXIT
logdir=$(mktemp -d)

# Start specified action for each project
declare -A pid_pro_map=() pid_log_map=()
readarray -t files < <(find . -name '.clasp.json' -printf "%P\n" | sort -V)
for file in "${files[@]}"; do
    project=$(dirname "${file}")
    logfile=$(mktemp -p "${logdir}")
    ( cd "${project}" && clasp "${action}" "${args[@]}" ) &>"${logfile}" &
    pid=$!; pid_pro_map[${pid}]="${project}"; pid_log_map[${pid}]="${logfile}"
    echo -e "Started action '\e[1m${action}\e[0m' for project '\e[1m${project}\e[0m' (pid ${pid})"
done

# Wait for background jobs to finish and report results
echo -e "\nWaiting for background jobs to finish...\n"
jobs_done=0; jobs_total=${#files[@]}
while true; do
    wait -n -p pid; result=$?
    [[ -z "${pid}" ]] && break
    jobs_done=$((jobs_done + 1))
    if (( ${result} == 0 )); then
        echo -e "Action '\e[1m${action}\e[0m' for project '\e[1m${pid_pro_map[${pid}]}\e[0m' (pid ${pid}) (${jobs_done}/${jobs_total}): \e[1;32mSUCCESS\e[0m"
    else
        echo -e "Action '\e[1m${action}\e[0m' for project '\e[1m${pid_pro_map[${pid}]}\e[0m' (pid ${pid}) (${jobs_done}/${jobs_total}): \e[1;31mFAILURE\e[0m"
        cat "${pid_log_map[${pid}]}"
    fi
done

Features:

• Allows to run any action supported by clasp (e.g. pull, push, deploy)
• Performs the specified action for each project in parallel in the background
• Output produced by clasp is suppressed (but captured to be printed in case of failure)
• Waits for background tasks to finish and reports results as soon as they become available
• Provides information regarding success/failure for each project (including output produced by clasp for further analysis in case of failure)
• Displays current progress (in the form of <projects-done>/<projects-total>)
• Colored output for increased readability

Requirements:

• Bash >= 5.1 (details: Bash >= 5.1 for wait -p, Bash >= 4.3 for wait -n, Bash >= 4.0 for associative arrays)
• GNU find (part of findutils) for find ... -printf "%P\n"; Possible workaround:

  readarray -t files < <(find . -name '.clasp.json' | sort -V)
  for file in "${files[@]}"; do
      project=$(dirname "${file#'./'}")
  

Sample output:

In response to this comment, here is a possible tweak to limit the amount of concurrent background jobs being spawned:

# Start specified action for each project
max_jobs=25; poll_delay="0.1s"
declare -A pid_pro_map=() pid_log_map=()
readarray -t files < <(find . -name '.clasp.json' -printf "%P\n" | sort -V)
for file in "${files[@]}"; do
    if (( ${max_jobs} > 0 )); then
        while jobs=$(jobs -r -p | wc -l) && (( ${jobs} >= ${max_jobs} )); do
            sleep "${poll_delay}"
        done
    fi
    project=$(dirname "${file}")
    logfile=$(mktemp -p "${logdir}")
    ( cd "${project}" && clasp "${action}" "${args[@]}" ) &>"${logfile}" &
    pid=$!; pid_pro_map[${pid}]="${project}"; pid_log_map[${pid}]="${logfile}"
    echo -e "Started action '\e[1m${action}\e[0m' for project '\e[1m${project}\e[0m' (pid ${pid})"
done

Additionally, this could be employed to cut the amount of background processes being spawned in half:

( cd "${project}" && exec clasp "${action}" "${args[@]}" ) &>"${logfile}" &

This will replace the subshell's process with clasp, which should be perfectly fine as the subshell looses its usefulness right after executing cd anyway.

1. The script does not cause a new shell prompt to appear during the execution of the script.

The new shell prompt is occurring because you are creating a new subshell in the while loop (for further guidance on how subshells work in bash, reference this page from tldp.org: link). To prevent this from occurring, call the command directly without placing them within parentheses.

2. The script outputs a line indicating the success or failure of each clasp pull operation, referenced by the directory name of the project (where the .clasp.json file was found).

You can generally catch if a command fails by adding an || after the command (e.g. grep "foobar" file.txt || echo "Error: 'foobar' not found in file.txt"). You could also put the command in an if/else and echo the appropriate status message for each.

3. Bonus: suppress the output of clasp pull so the script only shows the success or failure result of each project (referenced by the directory name).

Note: This response uses the aforementioned solution from the second question. You could create 2 arrays—1 for success and 1 for failures, and then inside of the if/else statements, add the current iteration element to the correct array.

Feel free to ask for clarification if any part of the above was not clear!

You should enforce script to wait for output before finish:

{ 
    while IFS= read -d $'\0' -ru $find file; do
        (
            cd "$(dirname "$file")"
            project_dir_name="$(basename "$(pwd)")"
            echo "Pulling project ($project_dir_name)"
            if clasp pull </dev/null 2>&1 ;then
                printf '\nExeClaspResult: %s Success\n' "$project_dir_name"
            else
                printf '\nExeClaspResult: %s Failed\n' "$project_dir_name"
            fi
        ) &
    done {find}< <(find . -name '.clasp.json' -print0)
    wait
} |
    sed -une 's/^ExeClaspResult: //p'

Where:

1. No interaction
2. All ouptut wll be dropped (by sed)
3. Only result will be shown.

If you wanna do some debugging:

{ 
    while IFS= read -d $'\0' -ru $find file; do
        (
            cd "$(dirname "$file")"
            project_dir_name="$(basename "$(pwd)")"
            echo "Pulling project ($project_dir_name)"
            if clasp pull </dev/null > >(
                  sed "s/^/OUT $project_dir_name: /") 2> >(
                  sed "s/^/ERR $project_dir_name: /"
                );then
                printf '\nExeClaspResult: %s Success\n' "$project_dir_name"
            else
                printf '\nExeClaspResult: %s Failed\n' "$project_dir_name"
            fi
        ) &
    done {find}< <(find . -name '.clasp.json' -print0)
    wait
} |
    sed -ue '
        s/^ExeClaspResult: \(.* Failed\)$/\o33[31m** \1 **\o33[0m/;
        s/^ExeClaspResult: \(.*\)$/\o33[32m** \1 **\o33[0m/;'

Will show all output prefixed by OUT $project or ERR project, and color results in green if success or red when failed.

My approach would be to log all the asynchronous invocations first. These will happen quickly. Then start printing the task failed/succeeded log.

This script uses named pipes to achieve this. Asynchronous commands happen in subshells, which inherit open file descriptors that the success/fail messages can print too. We can wait until all the invocations are logged, before sending these to the terminal.

Output of pull all is hidden. It can be replaced with another command or shell function.

stdout and stderr can be redirected as normal (like pull-all 2>err-log, or pull-all >/dev/null to see errors only).

The script waits for the pull commands to complete, then exits normally.

If it's interrupted (ctrl+c), all child processes are killed. Otherwise they would keep running after the script exits. I'm not sure if there's a better way to handle that or not.

Solution:

#!/bin/bash

cleanup () {
    exec 3>&-
    exec 4>&-

    rm .pull-all-log.fifo .pull-all-log-err.fifo

    kill $(jobs -p)
    kill -9 $(jobs -p) &>/dev/null
}

# use Node 14.17.5 to prevent "Error: Looks like you are offline." errors
# (see https://github.com/google/clasp/issues/872)
[[ -s "/usr/local/opt/nvm/nvm.sh" ]] && . "/usr/local/opt/nvm/nvm.sh"
nvm install 14.17.5
nvm use 14.17.5

# you can use /tmp or mktemp -u if you're worried about clobbering
rm -f .pull-all-log.fifo .pull-all-log-err.fifo
mkfifo .pull-all-log.fifo .pull-all-log-err.fifo
trap cleanup EXIT

exec 3<> .pull-all-log.fifo
exec 4<> .pull-all-log-err.fifo

for file in ./*/.clasp.json; do
    [[ -d "$file" ]] && continue

    parent=$(dirname "$file")
    proj=${parent##*/}

    echo "Pulling $proj..."

    {
        cd "$parent"

        if clasp pull &>/dev/null; then
            echo "Pulling $proj succeeded" >&3
        else
            echo "Pulling $proj FAILED" >&4
        fi
    } &
done

running_pids=$(jobs -p)

cat <&3 &
cat <&4 >&2 &

[[ "$running_pids" ]] && wait $running_pids

Example output:

Pulling project-1...
Pulling project-2...
Pulling project-3...
Pulling project-4...
Pulling project-5...
Pulling project-6...
Pulling project-7...
Pulling project-8...
Pulling project-9...
Pulling project-1 succeeded
Pulling project-9 succeeded
Pulling project-3 succeeded
Pulling project-4 FAILED
Pulling project-5 succeeded
Pulling project-2 succeeded
Pulling project-6 FAILED
Pulling project-7 FAILED
Pulling project-8 FAILED

FAILED or succeeded are printed for every invocation, whenever it finishes (even if it's before the invocation loop ends).

The example directories are named project-1 etc. I recreated your example tree to test it.

Can you do:

parallel -j100 --tag 'cd {//} && clasp pull || echo Fail' ::: */.clasp.json

If you only want output from clasp if it fails:

parallel -j100 --tag 'cd {//} && neno clasp pull || echo Fail' ::: */.clasp.json

neno is https://gitlab.com/ole.tange/tangetools/-/tree/master/neno