NGINX Ingress times out with large file downloads

Viewed 773

We are running a Spring Boot app in a k8s pod that is hosted behind an NGINX ingress with a EC2 load balancer. Our app occasionally needs to send a very large file (10/20 GB). We have observed that this operation occasionally times out when querying through the ingress, but does not timeout when queried directly. To more easily reproduce this, we created a simple endpoint to request a file of arbitrary size (/files/SIZE). That is what you can see below.

When a request times out, the ingress controller does not seem to post any logs. From the HTTP client, when the request times out, here is what we are given:

{ [3744 bytes data]
100 16.4G    0 16.4G    0     0  22.7M      0 --:--:--  0:12:23 --:--:-- 23.9M* TLSv1.2 (IN), TLS alert, close notify (256):
{ [2 bytes data]
100 16.5G    0 16.5G    0     0  22.7M      0 --:--:--  0:12:23 --:--:-- 23.6M
* Connection #0 to host INGRESS_URL left intact
* Closing connection 0
curl INGRESS_URL/files/21474836480 -v    31.47s user 26.92s system 7% cpu 12:23.81 total

Here is the configuration of our ingress:

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: USER
  namespace: NAMESPACE
  selfLink: /apis/extensions/v1beta1/namespaces/NAMESPACE/ingresses/USER
  uid: d84f3ab2-7f2c-42c1-a44f-c6a7d432f03e
  resourceVersion: '658287365'
  generation: 1
  creationTimestamp: '2021-06-29T13:21:45Z'
  labels:
    app.kubernetes.io/instance: USER
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: APP
    helm.sh/chart: CHART
  annotations:
    kubernetes.io/ingress.class: nginx-l4-ext
    meta.helm.sh/release-name: USER
    meta.helm.sh/release-namespace: NAMESPACE
    nginx.ingress.kubernetes.io/client-max-body-size: '0'
    nginx.ingress.kubernetes.io/proxy-body-size: '0'
    nginx.ingress.kubernetes.io/proxy-buffering: 'off'
    nginx.ingress.kubernetes.io/proxy-max-temp-file-size: '0'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '1800'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '1800'
    nginx.ingress.kubernetes.io/websocket-services: core-service
    nginx.org/websocket-services: core-service
  managedFields:
    - manager: Go-http-client
      operation: Update
      apiVersion: networking.k8s.io/v1beta1
      time: '2021-06-29T13:21:45Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubernetes.io/ingress.class': {}
            'f:meta.helm.sh/release-name': {}
            'f:meta.helm.sh/release-namespace': {}
            'f:nginx.ingress.kubernetes.io/client-max-body-size': {}
            'f:nginx.ingress.kubernetes.io/proxy-body-size': {}
            'f:nginx.ingress.kubernetes.io/proxy-buffering': {}
            'f:nginx.ingress.kubernetes.io/proxy-max-temp-file-size': {}
            'f:nginx.ingress.kubernetes.io/proxy-read-timeout': {}
            'f:nginx.ingress.kubernetes.io/proxy-send-timeout': {}
            'f:nginx.ingress.kubernetes.io/websocket-services': {}
            'f:nginx.org/websocket-services': {}
          'f:labels':
            .: {}
            'f:app.kubernetes.io/instance': {}
            'f:app.kubernetes.io/managed-by': {}
            'f:app.kubernetes.io/name': {}
            'f:helm.sh/chart': {}
        'f:spec':
          'f:rules': {}
    - manager: nginx-ingress-controller
      operation: Update
      apiVersion: networking.k8s.io/v1beta1
      time: '2021-06-29T13:21:59Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:loadBalancer':
            'f:ingress': {}
spec:
  rules:
    - host: HOST_URL.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              serviceName: SERVICE_NAME
              servicePort: 9081
status:
  loadBalancer:
    ingress:
      - hostname: LOAD_BALANCER_URL

We are running ingress-nginx@v0.46.0

If anyone has any suggestions for why our large downloads are timing out, that would be great!

Testing Already Done:

  1. Verified the params are actually appearing in the generated nginx.conf
  2. Tried changing client-body-timeout - this had to effect.
  3. Recreated the whole environment on my local minikube instance. The application works there. Is it possible this is an Amazon ELB issue?
  4. Changing spring.mvc.async.request-timeout does not fix the issue.
  5. The issue only occurs when making HTTPS calls. HTTP calls run totally fine
1 Answers

I had a similar issue with one of my SpringBoot Apps and the issue was with the Springboot configuration in the application.properties file.

spring:
  mvc:
    async:
      request-timeout: 3600000

Reference: https://stackoverflow.com/a/43496244/2777988

Related