Parallel execution of shell processes

Viewed 82763

Is there a tool available to execute several process in parallel in a Windows batch file? I have found some interesting tools for Linux (parallel and PPSS), however, I would need a tool for Windows platforms.

Bonus: It would be great if the tool also allowed to distribute processes in an easy way among several machines, running the processes remotely a la PsExec.

Example: I would like that in the following for loop

for %F in (*.*) do processFile.exe %F

a limited amount of instances of processFile.exe are running in parallel to take advantage of multi-core CPUs.

6 Answers

Try start:

start "title of the process" "P:\ath\to.exe"

It opens a new window with the given title and executes the BAT, CMD or EXE file. You can also set the priority, set the same environment etc.

Files being not executeable are opened with the associated program.

Further reading: Start -> Run

cmd /k start /?

Start is available at least since WinME.

Good luck!

Sounds more like you want to use Powershell 2. However, you can spawn new cmd windows (or other processes) by using start, see also this answer. Although you probably have to use some other tools and a little trickery to create something like a "process pool" (to have only a maximum of n instances running at a time). You could achieve the latter by using tasklist /im and counting how many are already there (for loop or wc, if applicable) and simply wait (ping -n 2 ::1 >nul 2>&1) and re-check again whether you can spawn a new process.

I have cobbled together a little test batch for this:

@echo off
for /l %%i in (1,1,20) do call :loop %%i
goto :eof

:loop
call :checkinstances
if %INSTANCES% LSS 5 (
    rem just a dummy program that waits instead of doing useful stuff
    rem but suffices for now
    echo Starting processing instance for %1
    start /min wait.exe 5 sec
    goto :eof
)
rem wait a second, can be adjusted with -w (-n 2 because the first ping returns immediately;
rem otherwise just use an address that's unused and -n 1)
echo Waiting for instances to close ...
ping -n 2 ::1 >nul 2>&1
rem jump back to see whether we can spawn a new process now
goto loop
goto :eof

:checkinstances
rem this could probably be done better. But INSTANCES should contain the number of running instances afterwards.
for /f "usebackq" %%t in (`tasklist /fo csv /fi "imagename eq wait.exe"^|find /c /v ""`) do set INSTANCES=%%t
goto :eof

It spawns a maximum of four new processes that execute in parallel and minimized. Wait time needs to be adjusted probably, depending on how much each process does and how long it is running. You probably also need to adjust the process name for which tasklist is looking if you're doing something else.

There is no way to properly count the processes that are spawned by this batch, though. One way would be to create a random number at the start of the batch (%RANDOM%) and create a helper batch that does the processing (or spawns the processing program) but which can set its window title to a parameter:

@echo off
title %1
"%2" "%3"

This would be a simple batch that sets its title to the first parameter and then runs the second parameter with the third as argument. You can then filter in tasklist by selecting only processes with the specified window title (tasklist /fi "windowtitle eq ..."). This should work fairly reliable and prevents too many false positives. Searching for cmd.exe would be a bad idea if you still have some instances running, as that limits your pool of worker processes.

You can use %NUMBER_OF_PROCESSORS% to create a sensible default of how many instances to spawn.

You can also easily adapt this to use psexec to spawn the processes remotely (but wouldn't be very viable as you have to have admin privileges on the other machine as well as provide the password in the batch). You would have to use process names for filtering then, though.

GNU xargs under Linux has a "-P n" switch to launch "n" processes in parallel.

Maybe cygwin/mingw build of xargs also supports this?

Then you can use:

xargs -P 4 processFile < fileList

No fancy multi-node process spawning, though.

I wrote a library which provides multithreading support (in a way that "emulates" the behavior of thread-pools)

MultiBat on github

Here is the inline version of that library but I'd suffest going to the link above for the latest version (and a version that doesn't require being inlined into your bat file.

REM ---------------------------------------------------------------------------
REM ---------------------------------------------------------------------------
REM ---------------------------------------------------------------------------
goto:EOF
REM Append this to the END of your batch-file [*.BAT] to get inline "Multi" support

REM  "Multi" is a thread-pool emulation helper library for controlling multi-threaded windows batch [*.BAT] files
REM  Copyright (c) 2020 Adisak Pochanayon
REM  Contact: adisak@gmail.com
REM  See Multi_License.txt for details

REM -----------------------------------

:Multi_Setup

call :Multi_SetName %1

if "%2"=="" (
    if "%NUMBER_OF_PROCESSORS%"=="" call :Multi_SetLimitToMax
) else (
    call :Multi_SetLimit %2
)
goto:EOF

REM -----------------------------------

:Multi_SetName
REM Returns: MULTI_CHILDPROC_WINNAME - name to use for child processes (the window title)

if "%1"=="" (
    SET MULTI_CHILDPROC_WINNAME=Multi-CmdProc
) else (
    SET MULTI_CHILDPROC_WINNAME=Multi-CmdProc-%1
)
goto:EOF

REM -----------------------------------

REM To Enable Hyperthreading, call Multi_SetHyperThread before calling Multi_Setup or Multi_SetLimitToMax

:Multi_SetHyperThread
REM Parameter 1: (optional)
REM        value=1    (or unspecified) - Use Hyperthreading if available
REM        value=0 (or other) - Do not use Hyperthreading to compute the max threads
REM Returns: NumberOfCores - number of real CPU cores
REM Returns: MULTI_HAS_HYPERTHREADING - 1 if the CPU has Hyperthreading
REM Returns: MULTI_USE_HYPERTHREADING - 1 if "Multi" should use Hyperthreading

REM Set variable NumberOfCores
if "%NumberOfCores%"=="" (
    for /f "tokens=*" %%f in ('wmic cpu get NumberOfCores /value ^| find "="') do set %%f
)

REM Set variable MULTI_HAS_HYPERTHREADING
if "%MULTI_HAS_HYPERTHREADING%"=="" (
    if "%NumberOfCores%"=="%NUMBER_OF_PROCESSORS%" (
        REM Non-Hyperthreading
        SET MULTI_HAS_HYPERTHREADING=0
    ) else (
        REM Hyperthreading
        SET MULTI_HAS_HYPERTHREADING=1
    )
}

if "%1"=="" (
    SET MULTI_USE_HYPERTHREADING=%MULTI_HAS_HYPERTHREADING%
) else (
    SET MULTI_USE_HYPERTHREADING=%1
)

REM Set the max threads to the limit (respecting Hyperthreading options)
call :Multi_SetLimitToMax
goto:EOF

REM -----------------------------------

:Multi_SetLimit
REM Parameter 1:
REM        value=N    - Use N as the number of max threads
REM        unspecified - Compute the default number of max threads
REM Returns: MULTI_MAXCHILDREN - the maximum number of child processes to run simultaneously

if "%1"=="" (
    if "%MULTI_MAXCHILDREN%"=="" call :Multi_SetLimitToMax
    goto:EOF
)

SET /A MULTI_MAXCHILDREN=%1
if %MULTI_MAXCHILDREN% LSS 1 SET MULTI_MAXCHILDREN=1
goto:EOF

REM -----------------------------------

:Multi_SetLimitToMax
REM Parameter 1: (optional)
REM        Passed to Multi_SetHyperThread if present
REM Returns: MULTI_MAXCHILDREN - max number of "threads" in pool for "Multi"

if "%1"=="" (
    REM Check if Hyperthreading support was initialized
    if "%NumberOfCores%"=="" (
        call :Multi_SetHyperThread 0
        REM Multi_SetHyperThread calls back to this subroutine so exit to prevent recursion
        goto:EOF
    )
) else (
    call :Multi_SetHyperThread %1
    REM Multi_SetHyperThread calls back to this subroutine so exit to prevent recursion
    goto:EOF
)

if %NUMBER_OF_PROCESSORS% LEQ 3 (
    SET MULTI_MAXCHILDREN=1
) else (
    if "%NumberOfCores%"=="%NUMBER_OF_PROCESSORS%" (
        REM Non-Hyperthreading
        SET /A MULTI_MAXCHILDREN=%NUMBER_OF_PROCESSORS%-2
    ) else if "%MULTI_USE_HYPERTHREADING%"=="1" (
        REM Hyperthreading available and used
        SET /A MULTI_MAXCHILDREN=%NUMBER_OF_PROCESSORS%/2 - 1
    ) else (
        REM Hyperthreading available but not used
        SET /A MULTI_MAXCHILDREN=%NUMBER_OF_PROCESSORS%-2
    )
)
goto:EOF

REM -----------------------------------

:Multi_RunWin

if "%MULTI_CHILDPROC_WINNAME%"=="" call :Multi_SetName

call :Multi_WaitChildrenMax
start "%MULTI_CHILDPROC_WINNAME%" /BELOWNORMAL cmd /c %*
goto:EOF

REM -----------------------------------

:Multi_RunWinMin

if "%MULTI_CHILDPROC_WINNAME%"=="" call :Multi_SetName

call :Multi_WaitChildrenMax
start "%MULTI_CHILDPROC_WINNAME%" /MIN /BELOWNORMAL cmd /c %*
goto:EOF

REM -----------------------------------

:Multi_RunSyncMin

REM Use this command to run things that mess with the window title
REM and otherwise would screw up the "Multi" System
start "Multi-Sync" /MIN /WAIT cmd /c %*
goto:EOF

REM -----------------------------------

:Multi_WaitChildrenMax

REM Wait until less than MULTI_MAXCHILDREN children are running

if "%MULTI_MAXCHILDREN%"=="" call :Multi_SetLimitToMax

call :Multi_WaitChildren %MULTI_MAXCHILDREN%
goto:EOF

REM -----------------------------------

:Multi_WaitChildren

SETLOCAL
REM multi_WAITCOUNT is a local variable
SET multi_WAITCOUNT=1

if "%1"=="" GOTO :loop_WaitChildren
SET /A multi_WAITCOUNT=%1
if %multi_WAITCOUNT% LSS 1 set multi_WAITCOUNT=1

:loop_WaitChildren
call :Multi_GetNumChildren
if %MULTI_NUM_CHILDREN% LSS %multi_WAITCOUNT% GOTO :exit_WaitChildren
timeout /t 1 /nobreak > nul
GOTO :loop_WaitChildren

:exit_WaitChildren
ENDLOCAL
goto:EOF

REM -----------------------------------

:Multi_GetNumChildren
REM Returns: MULTI_NUM_CHILDREN - the number of "children" processes (Windows named MULTI_CHILDPROC_WINNAME)

if "%MULTI_CHILDPROC_WINNAME%"=="" call :Multi_SetName

REM MULTI_NUM_CHILDREN should contain the number of 
REM running %MULTI_CHILDPROC_WINNAME% instances after this
for /f "usebackq" %%t in (`tasklist /fo csv /fi "WINDOWTITLE eq %MULTI_CHILDPROC_WINNAME%" ^| find /c "cmd"`) do (
    SET MULTI_NUM_CHILDREN=%%t
)
goto:EOF

REM -----------------------------------
Related