I am trying to write a small application using the libcurl multi interface in C++ and am running into an issue where it performs (much) slower than expected: for 49 requests (arbitrary number) against a static HTML page (served by nginx on a small VM I rent on the internet) it takes a pretty constant 32 seconds.
Running the application against Google or GitHub takes just as long and benching my static page using hey yields >1500 requests/s, so I'm pretty sure it's not the servers fault. Looping over the curl command line application 50 times in a row is also quite a bit faster with just 13 seconds total.
Here is the code in question (minimal code for readability):
#include <curl/curl.h>
#include <memory>
#include <string>
#include <vector>
struct Request {
std::shared_ptr<CURL> request;
};
int request(std::string url, CURLM *curlm, std::vector<Request> &requests) {
Request r = {
.request = std::shared_ptr<CURL>(curl_easy_init(), curl_easy_cleanup),
};
curl_easy_setopt(r.request.get(), CURLOPT_URL, url.c_str());
curl_multi_add_handle(curlm, r.request.get());
requests.push_back(r);
return 0;
}
int main(__attribute__((unused)) int argc, char *argv[]) {
curl_global_init(CURL_GLOBAL_ALL);
std::vector<Request> requests;
CURLM *multi = curl_multi_init();
// generate an arbitrary amount of requests
for(int i = 0; i < 49; i++)
request(std::string(argv[1]), multi, requests);
int running = 0;
curl_multi_perform(multi, &running);
do {
int numfds = 0;
curl_multi_perform(multi, &running);
// select
fd_set fdread, fdwrite, fdexcep;
FD_ZERO(&fdread);
FD_ZERO(&fdwrite);
FD_ZERO(&fdexcep);
struct timeval timeout = {
.tv_sec = 0,
.tv_usec = 500
};
curl_multi_fdset(multi, &fdread, &fdwrite, &fdexcep, &numfds);
select(numfds+1, &fdread, &fdwrite, &fdexcep, &timeout);
// or poll
//curl_multi_poll(multi, nullptr, 0, 1000, &numfds);
} while(running > 0);
return 0;
}
I have tested on multiple client systems: my workstation is an Ubuntu 20.04, the results can be reproduced on CentOS 8, Ubuntu 18.04 and Debian 10 (all libcurl 7.60 series), however on CentOS 7 (libcurl 7.29.0) the same code finishes within 3-5 seconds! Apart from my workstation all of these systems I tested are fresh installed VMs running on the same host with the same network configuration. Meanwhile, the VM on the internet (Debian 10, libcurl 7.64) is about as fast as I expect with sub-second results.
Using Wireshark I can see all connections to the webserver are opened at once, but the actual HTTP requests are sent in batches with increasing intervals - the last ones are sent as late as ~30 seconds after opening the connection.
There is no time difference between the select() way of doing things vs using curl_multi_poll() either and I have gone through the cURL documentation looking for options that might increase performance; enabling pipelining does not make any difference nor does disabling HTTP2 and most of the limits (max connections, for example) seem to be set to unlimited by default.
Now here is the fun part: when I run this application through strace, a run only takes 2-4 seconds per run on any of the local VMs. At first I thought strace might be breaking things, however if I discard the output of strace via shell redirection it looks like the application is working properly, so what gives?
Why does using strace improve things at all and what may cause run time to be wildly different across these systems?
Any ideas what I'm missing? My Google-fu has left me this time.