We observe that if we reload the nginx configuration the in-flight HTTP 1.1 requests complete successfully but without Connection:close which cause clients to attempt to re-use a closed connection. The behaviour can be reproduced easily by doing a curl with two requests where during the first request we reload the nginx configuration by HUP signal.
curl -v -H "Host: foo.bar.host" "http://proxy1:4080/do-it-takes-time-and-reload-is-called" "http://proxy1:4080/do-it"
* Connected to proxy1 (xx.xxx.xxx.xxx) port 4080 (#0)
>GET /do-it-takes-time-and-reload-is-called HTTP/1.1
> Host: foo.bar.host
> User-Agent: curl/7.54.0
> Accept: */*
< HTTP/1.1 200 OK
< Server: nginx/1....
< Date: ..
< Content-Type: application/json;charset=utf-8
< Content-Length: 160
< Connection: keep-alive
< Vary: Accept-Encoding
< Keep-Alive: timeout=10, max=360
* Connection #0 to host proxy left intact
{ "foo":"bar"}
While this long lived request was in-fight we did a kill -HUP to reload the configuration but notice how the response says Connection:keep-alive so curl attempts to re-use the connection but server don't want any of that so the request is rejected (RST)
* Found bundle for host proxy1 0x7f96e980f510 [can pipeline]
* Re-using existing connection! (#0) with host proxy1
* Connected to proxy1 (xx.xxx.xxx.xxx) port 4080 (#0)
> GET /do-it HTTP/1.1
> Host: foo.bar.host
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection died, retrying a fresh connect
* Closing connection 0
* Issue another request to this URL: ..
So is there a way to make nginx end the conversation politely with a Connection:close in the event of a configuration reload?