We recently upgraded from Spring boot 2.1.2 Release to Spring boot 2.2.0 Release and started having an issue "502 error upstream sent too big header" from upstream service. Here is the request flow:
Browser --> K8s nginx (1.15.0) --> api-gateway(Zuul with Spring Boot 2.2) --> Backend service.
Backend response is a 302 redirect with location that has URL and some app data.
Only thing that is changed here is Spring boot upgrade in api-gateway and backend services.
I verified responses ( headers and data) from the api-gateway to the nginx k8s server and responses are same between Spring boot versions.
Here is the error in nginx logs: 2021/09/29 17:16:01 [error] 44#44: *68665904 upstream sent too big header while reading response header from upstream, client: 10.221.200.342, server:gateway.x.y.io, request: "POST /r/l HTTP/2.0", upstream: "http://10.221.200.342:8080/x/y", host: "gateway.x.y.io"
Upgraded packages:
Old: Spring boot 2.1.3 Release, org.springframework.cloud:spring-cloud-dependencies:Greenwich.RELEASE and Standard spring boot rest dependencies.
New: Spring boot 2.2.0 Release, org.springframework.cloud:spring-cloud-dependencies:Hoxton.RELEASE and Standard spring boot rest dependencies
Appreciate suggestions on how to debug the root cause or resolve this issue other than increasing proxy_buffer limits in nginx .
--10/6 update
More updates: Here are logs from nginx access logs for successful response. Reduced location length in response so it succeeds in Spring 2.2.
Spring Boot 2.2.0 - 10.244.2.128 - [10.244.2.128] - - [06/Oct/2021:01:35:56 +0000] "POST /x/y HTTP/2.0" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.54 Safari/537.36" 7580 3.354 [x-y-z-80] 10.244.1.48:8080 0 3.290 302 7d9cfce0f1ae70b1ec007a9474f60b9e
Spring Boot 2.1.2 - 10.244.2.128 - [10.244.2.128] - - [06/Oct/2021:13:16:34 +0000] "POST /x/y HTTP/2.0" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.54 Safari/537.36" 7580 3.091 [z-x-y-80] 10.244.1.97:8080 0 3.026 302 62f2348644328eedacad8ccc9b195415.
Thing to notice: texts in bold are the response length ($upstream_response_length nginx) and are different.