Is there a way to accomplish multithreading or parallel processes in a batch file?

Question

So I have a batch file that is executing a simulation given some input parameters and then processing the output data via awk, R, and Python. At the moment the input parameters are passed into the simulation through some nested for loops and each iteration of the simulation will be run one after the other. I would like for the execution of the simulation to be done in parallel because at the moment there are 1,000+ cases so in my mind I could have core 1 handle sims 1-250, core 2 handle sims 251-500, etc.

In essence what I would like to do is this:

Run every case of the simulation across multiple cores
Once every simulation has been completed, start the output data processing

I've tried using start /affinity n simulation.exe, but the issue here is that all of the simulations will be executed simultaneously, so when it gets to the post processing calls, it errors out because the data hasn't been generated yet. There is the start /w command, but I'm not sure if that improve the simulation. One idea I've thought of is updating a variable once each simulation has been completed, then only start the post processing once the variable reaches n runs.

Here is an excerpt of what I am doing right now:

    for %%f in (1 2 3) do (
            for %%a in (4 5 6) do (
                for %%b in (7 8 9) do (
                    call :mission %%f %%a %%b
                )
            )
         )
    some gawk scripts
    some python scripts
    some r scripts
    go to :exit

:mission
   sed -e 's/text1/%1/' -e 's/text2/%2/' -e 's/text3/%3/'
   simulation.exe
   go to :exit

:exit

And here's what I was playing around with to test out some parallel processing:

start /affinity 1 C:\Users\614890\R-4.1.1\bin\Rscript.exe test1.R
start /affinity 2 C:\Users\614890\R-4.1.1\bin\Rscript.exe test2.R
start /affinity 3 C:\Users\614890\R-4.1.1\bin\Rscript.exe test3.R
start /affinity 4 C:\Users\614890\R-4.1.1\bin\Rscript.exe test4.R

C:\Users\614890\R-4.1.1\bin\Rscript.exe plotting.R

user_16588266 · Answer

I was actually able to accomplish this by doing the following:

setlocal
set "lock=%temp%\wait%random%.lock"

:: Launch processes asynchronously, with stream 9 redirected to a lock file.
:: The lock file will remain locked until the script ends
start /affinity 1 9>"%lock%1" Rscript test1.R
start /affinity 2 9>"%lock%2" Rscript test2.R
start /affinity 4 9>"%lock%3" Rscript test3.R
start /affinity 8 9>"%lock%4" Rscript test4.R

:Wait for all processes to finish
1>nul 2>nul ping /n 2 ::1
for %%F in ("%lock%*") do (
 (call ) 9>"%%F" || goto :Wait
) 2>nul

del "%lock%*"

Rscript plotting.R

Is there a way to accomplish multithreading or parallel processes in a batch file?

1 Answers