Parallel processing or multi-threading in Shell scripting

user_13068766 0
Viewed 138

I am writing a script in shell to pull files from application using curl call, and to pull 100 files it is taking more than 30 minutes.

I want split this into multiple chunks and do a parallel curl call (eg: 10 files each call).

I am new in parallel processing/threading.

parallel-processing shell multithreading scripting bash
1 Answers

Q : "I want ... do a parallel curl call ..."

For all possible syntax details, start with man parallel

Next, one may use also some curl-tricks, yet only those that do not collide with the parallel syntax-elements.

As the fileIO-ops are both slow and bear rather a high ( yet maskable ) latency, a number of concurrent processes might grow well high :

parallel --jobs 24 \
         --dry-run  \
    curl             \
         ftps://a.b.c.d/node7-{1}/{1}-{2}/{1}-{2}-{3}-{4}_*.jpg \
    ::: "LKLN" "LKRO" "LKPM"                                     \
    ::: $( seq -f "%04g" 2011 2020 )                              \
    ::: $( seq -f "%02g" 4  6 )                                    \
    ::: $( seq -f "%02g" 1 31 )                                     \

will yield a demo of :

...
curl ftps://a.b.c.d/node7-LKLN/LKLN-2020/LKLN-2020-06-29_*.jpg
curl ftps://a.b.c.d/node7-LKLN/LKLN-2020/LKLN-2020-06-30_*.jpg
curl ftps://a.b.c.d/node7-LKLN/LKLN-2020/LKLN-2020-06-31_*.jpg
curl ftps://a.b.c.d/node7-LKRO/LKRO-2011/LKRO-2011-04-01_*.jpg
curl ftps://a.b.c.d/node7-LKRO/LKRO-2011/LKRO-2011-04-02_*.jpg
curl ftps://a.b.c.d/node7-LKRO/LKRO-2011/LKRO-2011-04-03_*.jpg
...

all split among the said 24 parallel-orchestrated processes

Finally feel free to adapt your scripting strategy so as to meet and match your actual transport, storage, processor and memory capacities, logging and self-reporting needs.

And you became a next master of the parallel-orchestrated processing.

All credits, since 2007, go to Ole Tange!

user_36661970
Related