Kubernetes readiness check does not correctly set HTTP host header

user_2945922 0
Viewed 3363

I have a Django application running in a container that I would like to probe for readiness. The kubernetes version is 1.10.12. The settings.py specifies to only allow traffic from a specific domain:

ALLOWED_HOSTS = ['.example.net']

If I set up my probe without setting any headers, like so: 

      containers:
      - name: django
       readinessProbe:
          httpGet:
            path: /readiness-path
            port: 8003

then I receive a 400 response, as expected- the probe is blocked from accessing readiness-path:

Invalid HTTP_HOST header: '10.5.0.67:8003'. You may need to add '10.5.0.67' to ALLOWED_HOSTS.

I have tested that I can can successfully curl the readiness path as long as I manually set the host headers on the request, so I tried setting the Host headers on the httpGet, as partially documented here, like so: 

        readinessProbe:
          httpGet:
            path: /readiness-path
            port: 8003
            httpHeaders:
              - name: Host
                value: local.example.net:8003

The probe continues to fail with a 400.

Messing around, I tried setting the httpHeader with a lowercase h, like so: 

        readinessProbe:
          httpGet:
            path: /django-admin
            port: 8003
            httpHeaders:
              - name: host
                value: local.example.net:8003

Now, the probe actually hits the server, but it's apparent from the logs that instead of overwriting the HTTP_HOST header with the correct value, it has been appended, and fails because the combined HTTP_HOST header is invalid:

Invalid HTTP_HOST header: '10.5.0.67:8003,local.example.net:8003'. The domain name provided is not valid according to RFC 1034/1035

Why would it recognize the header here and add it, instead of replacing it?

One suspicion I am trying to validate is that perhaps correct handling of host headers was only added to the Kubernetes httpHeaders spec after 1.10. I have been unable to find a clear answer on when host headers were added to Kubernetes- there are no specific headers described in the API documentation for 1.10.

Is it possible to set host headers on a readiness probe in Kubernetes 1.10, and if so how is it done? If not, any other pointers for getting this readiness probe to correctly hit the readiness path of my application?

Update:

I have now tried setting the value without a port, as suggested by a commenter:

            httpHeaders:
              - name: Host
                value: local.acmi.net.au

The result is identical to setting the value with a port. With a capital H the host header value is not picked up at all, with a lowercase h the host header value is appended to the existing host header value.

kubernetes
2 Answers

This problem is fixed in Kubernetes 1.2.3+ — see kubernetes/kubernetes#24288. I've configured one of my deployments in a similar way (for similar reasons). This is for a Craft CMS instance which doesn't require authentication at the /admin/login url:

    readinessProbe:
      httpGet:
        path: /admin/login
        port: 80
        httpHeaders:
        - name: Host
          value: www.example.com
      timeoutSeconds: 5
user_4823820

Steven Shaw's answer was very useful, but the winning combination wound up finally being:

readinessProbe:
          httpGet:
            path: /readiness-path/ # << NO REDIRECTS ON THIS PATH
            port: 8003
            httpHeaders:
              - name: host
                value: local.example.net # << NO PORT ON THE HOST VALUE

After some log contemplation I finally realized that the path I was probing for readiness was redirecting from /readiness-path to /readiness-path/. Once I supplied the trailing slash to sidestep the redirect, it started working. From this I conclude that the redirects weren't preserving the httpHeaders being set on the probe request.

user_29459220
Related