I am getting a few 502 errors in nginx:
2020/05/05 20:40:05 [error] 13425#0: *1360447 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.31.12.13, server: , request: "POST /app/functionHTTP/1.1", upstream: "http://127.0.0.1:8080/app/function", host: "example.com"
Architecture:
client - load balancer - instances (each with nginx proxy for NodeJS http server)
Timeout values:
client:
- request timeout: 60s
load balancer:
- keepalive timeout: 60s
nginx:
- http keepalive_timeout: 65s
- upstream keepalive_timeout: 60s
- upstream keepalive: 256 connections
- proxy_connect_timeout: 60s
- proxy_read_timeout: 60s
- proxy_send_timeout: 60s
server (NodeJS):
- request timeout: 50s
- keepAliveTimeout: 65s
- headersTimeout: 70s
The 502 errors occur rarely (~0.01% of requests) but consistently.
What I've checked so far:
- Limit of file descriptors not reached
- Limit of nginx connections not reached
- Limit of TCP ports not reached
- Limit of CPU, memory, disk I/O not reached
It looks much like a race condition where the NodeJS server closed the connection and nginx is assuming the connection still exists.
What would be proper nginx timeout values in the current setting?
(I am unsure whether this isn't rather a question for ServerFault?)