We are running a Spring Boot app in a k8s pod that is hosted behind an NGINX ingress with a EC2 load balancer. Our app occasionally needs to send a very large file (10/20 GB). We have observed that this operation occasionally times out when querying through the ingress, but does not timeout when queried directly. To more easily reproduce this, we created a simple endpoint to request a file of arbitrary size (/files/SIZE). That is what you can see below.
When a request times out, the ingress controller does not seem to post any logs. From the HTTP client, when the request times out, here is what we are given:
{ [3744 bytes data]
100 16.4G 0 16.4G 0 0 22.7M 0 --:--:-- 0:12:23 --:--:-- 23.9M* TLSv1.2 (IN), TLS alert, close notify (256):
{ [2 bytes data]
100 16.5G 0 16.5G 0 0 22.7M 0 --:--:-- 0:12:23 --:--:-- 23.6M
* Connection #0 to host INGRESS_URL left intact
* Closing connection 0
curl INGRESS_URL/files/21474836480 -v 31.47s user 26.92s system 7% cpu 12:23.81 total
Here is the configuration of our ingress:
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: USER
namespace: NAMESPACE
selfLink: /apis/extensions/v1beta1/namespaces/NAMESPACE/ingresses/USER
uid: d84f3ab2-7f2c-42c1-a44f-c6a7d432f03e
resourceVersion: '658287365'
generation: 1
creationTimestamp: '2021-06-29T13:21:45Z'
labels:
app.kubernetes.io/instance: USER
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: APP
helm.sh/chart: CHART
annotations:
kubernetes.io/ingress.class: nginx-l4-ext
meta.helm.sh/release-name: USER
meta.helm.sh/release-namespace: NAMESPACE
nginx.ingress.kubernetes.io/client-max-body-size: '0'
nginx.ingress.kubernetes.io/proxy-body-size: '0'
nginx.ingress.kubernetes.io/proxy-buffering: 'off'
nginx.ingress.kubernetes.io/proxy-max-temp-file-size: '0'
nginx.ingress.kubernetes.io/proxy-read-timeout: '1800'
nginx.ingress.kubernetes.io/proxy-send-timeout: '1800'
nginx.ingress.kubernetes.io/websocket-services: core-service
nginx.org/websocket-services: core-service
managedFields:
- manager: Go-http-client
operation: Update
apiVersion: networking.k8s.io/v1beta1
time: '2021-06-29T13:21:45Z'
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:annotations':
.: {}
'f:kubernetes.io/ingress.class': {}
'f:meta.helm.sh/release-name': {}
'f:meta.helm.sh/release-namespace': {}
'f:nginx.ingress.kubernetes.io/client-max-body-size': {}
'f:nginx.ingress.kubernetes.io/proxy-body-size': {}
'f:nginx.ingress.kubernetes.io/proxy-buffering': {}
'f:nginx.ingress.kubernetes.io/proxy-max-temp-file-size': {}
'f:nginx.ingress.kubernetes.io/proxy-read-timeout': {}
'f:nginx.ingress.kubernetes.io/proxy-send-timeout': {}
'f:nginx.ingress.kubernetes.io/websocket-services': {}
'f:nginx.org/websocket-services': {}
'f:labels':
.: {}
'f:app.kubernetes.io/instance': {}
'f:app.kubernetes.io/managed-by': {}
'f:app.kubernetes.io/name': {}
'f:helm.sh/chart': {}
'f:spec':
'f:rules': {}
- manager: nginx-ingress-controller
operation: Update
apiVersion: networking.k8s.io/v1beta1
time: '2021-06-29T13:21:59Z'
fieldsType: FieldsV1
fieldsV1:
'f:status':
'f:loadBalancer':
'f:ingress': {}
spec:
rules:
- host: HOST_URL.com
http:
paths:
- path: /
pathType: Prefix
backend:
serviceName: SERVICE_NAME
servicePort: 9081
status:
loadBalancer:
ingress:
- hostname: LOAD_BALANCER_URL
We are running ingress-nginx@v0.46.0
If anyone has any suggestions for why our large downloads are timing out, that would be great!
Testing Already Done:
- Verified the params are actually appearing in the generated nginx.conf
- Tried changing client-body-timeout - this had to effect.
- Recreated the whole environment on my local minikube instance. The application works there. Is it possible this is an Amazon ELB issue?
- Changing
spring.mvc.async.request-timeoutdoes not fix the issue. - The issue only occurs when making HTTPS calls. HTTP calls run totally fine