connect() failed (113: Host is unreachable) while connecting to upstream nginx for aws

user_1294870 0
Viewed 5316

I know the this question is asked multiple times and not related to aws.

2020/07/29 10:23:17 [error] 6#6: *37749 connect() failed (113: Host is unreachable) while connecting to upstream, client:

I am facing this issue while I have deployed nginx in aws cloud.

localtion configuration

location /test {
         proxy_pass http://test-service;
         proxy_set_header HOST $host;
         proxy_set_header X-Forwarded-Proto $scheme;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         }

Upstream section like below

upstream test-service {
    server test-service.internal:38102;
    keepalive 10;
}

Here test-service.internal is my route53 hosted zone entry and it is pointing to some internal application load balancer of aws.

When I deploy/restart nginx server, it works well but after few days (around two/three days) it will hang in proxy pass only. When I load html content, it works perfectly but proxy pass call stuck.

Any solution would be helpful?

Thanks.

amazon-web-services nginx
1 Answers

After long debugging, we found that nginx will cache test-service.internal ips. And aws will chang it's internal load balancer's ips.

So nginx cached ips are no more exist. so we need to provide new ips.

Solution:

nginx has provided resolver directive

location /test {
           resolver 10.0.0.2 127.0.0.1 valid=30s;
           set $backend_servers test-service.internal;
           proxy_pass http://$backend_servers:38102;
         #proxy_pass http://test-service;
         proxy_set_header HOST $host;
         proxy_set_header X-Forwarded-Proto $scheme;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_http_version 1.1;
           proxy_set_header Connection "";
         }

We have changed two things.

  1. Added resolver.

  2. Removed upstream (resolver is not supported in nginx. nginx-plus support the upstream with resolver)

    resolver 10.0.0.2 127.0.0.1 valid=30s;    
set $backend_servers test-service.internal;
proxy_pass http://$backend_servers:38102;

Now we are using aws dns server 10.0.0.2 to resolve test-service.internal after every 30s

user_12948700
Related