Run multiple curl commands in parallel

Admin 1
Viewed 74188

I have the following shell script. The issue is that I want to run the transactions parallel/concurrently without waiting for one request to finish to go to the next request. For example if I make 20 requests, I want them to be executed at the same time.

for ((request=1;request<=20;request++))
do
    for ((x=1;x<=20;x++))
    do
        time curl -X POST --header "http://localhost:5000/example"
    done
done

Any guide?

curl shell bash
8 Answers

You can use xargs with -P option to run any command in parallel:

seq 1 200 | xargs -n1 -P10  curl "http://localhost:5000/example"

This will run curl command 200 times with max 10 jobs in parallel.

user_35545370

Using xargs -P option, you can run any command in parallel:

xargs -I % -P 8 curl -X POST --header "http://localhost:5000/example" \
< <(printf '%s\n' {1..400})

This will run give curl command 400 times with max 8 jobs in parallel.

user_5482250

Update 2020:

Curl can now fetch several websites in parallel:

curl --parallel --parallel-immediate --parallel-max 3 --config websites.txt

websites.txt file:

url = "website1.com"
url = "website2.com"
url = "website3.com"
user_134735250

This is an addition to @saeed's answer.

I faced an issue where it made unnecessary requests to the following hosts

0.0.0.1, 0.0.0.2 .... 0.0.0.N

The reason was the command xargs was passing arguments to the curl command. In order to prevent the passing of arguments, we can specify which character to replace the argument by using the -I flag.

So we will use it as,

 ... xargs -I '$' command ...

Now, xargs will replace the argument wherever the $ literal is found. And if it is not found the argument is not passed. So using this the final command will be.

seq 1 200 | xargs -I $ -n1 -P10  curl "http://localhost:5000/example"

Note: If you are using $ in your command try to replace it with some other character that is not being used.

user_70319330

Adding to @saeed's answer, I created a generic function that utilises function arguments to fire commands for a total of N times in M jobs at a parallel

function conc(){
    cmd=("${@:3}")
    seq 1 "$1" | xargs -n1 -P"$2" "${cmd[@]}"
}

$ conc N M cmd
$ conc 10 2 curl --location --request GET 'http://google.com/'

This will fire 10 curl commands at a max parallelism of two each.

Adding this function to the bash_profile.rc makes it easier. Gist

user_43555290

Add “wait” at the end, and background them.

for ((request=1;request<=20;request++))
do
    for ((x=1;x<=20;x++))
    do
        time curl -X POST --header "http://localhost:5000/example" &
    done
done

wait

They will all output to the same stdout, but you can redirect the result of the time (and stdout and stderr) to a named file:

time curl -X POST --header "http://localhost:5000/example" > output.${x}.${request}.out 2>1 &
user_86396060

Wanted to share my example how I utilised parallel xargs with curl.

The pros from using xargs that u can specify how many threads will be used to parallelise curl rather than using curl with "&" that will schedule all let's say 10000 curls simultaneously.

Hope it will be helpful to smdy:

#!/bin/sh

url=/any-url
currentDate=$(date +%Y-%m-%d)
payload='{"field1":"value1", "field2":{},"timestamp":"'$currentDate'"}'
threadCount=10

cat $1 | \
xargs -P $threadCount -I {} curl -sw 'url= %{url_effective}, http_status_code = %{http_code},time_total = %{time_total} seconds \n' -H "Content-Type: application/json" -H "Accept: application/json" -X POST $url --max-time 60 -d $payload

.csv file has 1 value per row that will be inserted in json payload

user_9194560

Based on the solution provided by @isopropylcyanide and the comment by @Dario Seidl, I find this to be the best response as it handles both curl and httpie.

# conc N M cmd - fire (N) commands at a max parallelism of (M) each
function conc(){
    cmd=("${@:3}")
    seq 1 "$1" | xargs -I'$XARGI' -P"$2" "${cmd[@]}"
}

For example:

conc 10 3 curl -L -X POST https://httpbin.org/post -H 'Authorization: Basic dXNlcjpwYXNz' -H 'Content-Type: application/json' -d '{"url":"http://google.com/","foo":"bar"}'

conc 10 3 http --ignore-stdin -F -a user:pass httpbin.org/post url=http://google.com/ foo=bar
user_20352070
Related