Edit: I found this to be a networking issue, but I don't have an answer on how to repair it yet, so hopefully someone else knows anything about it:
When i'm inside the Nginx container, I can query node.js like this:
curl http://192.168.65.2:3001/api/getTest
and that works, but has the same erratic behaviour as Nginx has below. So it indeed does mostly time-out on the backend for some networking reason I do not understand.
So when I run ab from the Nginx container:
> ab -n 10000 -c 5 http://192.168.65.2:3001/api/getTest
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.65.2 (be patient)
apr_pollset_poll: The timeout specified has expired (70007)
Total of 16 requests completed
Which indeed is the same behaviour as I saw from Nginx -> backend. Considering node.js works fine (seeing from the below ab on node.js directly; I even ran both simultaneously; the node.js version always finished with 0 errors, the ab from Docker never finishes correctly as shown above and below).
------- Old question, needed to understand the full case:
I have a standard nginx docker image:
image: nginx
port mapping:
- "8080:80"
I have a node server running outside docker with node v17.3.0 on port 3001.
I proxy pass requests from nginx 8080 to 3001 with the following configuration;
upstream backend {
server host.docker.internal:3001;
keepalive 32;
}
location /api {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://backend;
proxy_set_header Host $http_host;
}
When I do requests with curl or via the web frontend, it works on both 8080 and 3001 as expected. However, I noticed that on :8080, even with very little subsequent requests , Nginx hangs; average when I manually just try curl requests, after average 10 in a row Nginx hangs (it seems to be at around 10 subsequent request 'blocks' as ab hangs always at around 50 when concurrency=5, 30 when currency is 3 etc completed requests and manually, it always hangs after 10 completed requests). And it hangs longer than 30s, always.
So I get try:
> curl http://localhost:8080/api/getTest
<myjsonbody>
> curl http://localhost:3001/api/getTest
<myjsonbody>
But sometimes(!), simply:
> curl http://localhost:8080/api/getTest
And nothing for 30s+ seconds after which it sometimes does return the correct result and sometimes not:
> time curl http://localhost:8080/api/getTest
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.21.4</center>
</body>
</html>
0.01s user 0.01s system 0% cpu 1:00.04 total
Now I look at the node.js logs, when sending requests to nginx and the ones that hang do not arrive at node.js until 30-60s later or, in the latter case, it doesn't arrive at all at node.js.
So, to test I try the following ab version:
> ab -n 10000 -c 5 http://localhost:8080/api/getTest
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
apr_pollset_poll: The timeout specified has expired (70007)
Total of 48 requests completed
And, directly to node.js:
ab -n 10000 -c 5 http://localhost:3001/api/getTest
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software:
Server Hostname: localhost
Server Port: 3001
Document Path: /api/getTest
Document Length: 41017 bytes
Concurrency Level: 5
Time taken for tests: 2.965 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 412610000 bytes
HTML transferred: 410170000 bytes
Requests per second: 3372.14 [#/sec] (mean)
Time per request: 1.483 [ms] (mean)
Time per request: 0.297 [ms] (mean, across all concurrent requests)
Transfer rate: 135877.00 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 0 1 3.7 1 118
Waiting: 0 1 3.1 1 118
Total: 0 1 3.7 1 119
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 2
95% 2
98% 3
99% 4
100% 119 (longest request)
All requests succeed directly to node.js while nginx almost immediately hangs. I tried direct node.js with many more requests concurrently and in total; it simply never hangs. Nginx almost immediately always hangs.
Any ideas? I wouldn't know how to even debug it. I used Wireshark to trace the data and it goes into Nginx when it fails, but often doesn't actually gets sent to node.js (and then it gives the above gateway error) while when it does reach node.js, it is as:
- Request received by Nginx
- 40s nothing
- Request received by Node.js from Nginx
- Immediate response by Node.js to Nginx
- Nginx delivers endresult after 40s+some ms.
I tried a lot of configs but it doesn't fix the problem.
Could it be a networking problem? It feels like it but how would I find out further than what I have tried already?