Nginx gateway timeout without much load on server

Question

I am using NGINX and UWSGI to power the python/Django based API backend with load balancer attached to AWS auto scale groups, my servers works fine in routine, but sometimes start getting 504 or 502 from server once in a month or two constantly for a day or more. Load on my server is less than routine, memory usage is fine, but still get 502 or 504

Using ubuntu 14.0.4

Here is how my nginx configuration looks

user www-data;
worker_processes 2;
pid /run/nginx.pid;
events {
    worker_connections 2048;
    multi_accept on;
    use epoll;
}
worker_rlimit_nofile 40000;
http {
    ##
    # Basic Settings
    ##
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 50s;
keepalive_requests 1024;
    types_hash_max_size 2048;
    # server_tokens off;
    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;
    client_body_timeout 12;
    client_header_timeout 12;
    send_timeout 10;
    limit_conn_zone $binary_remote_addr zone=addr:10m;
    limit_conn addr 20;
    open_file_cache max=5000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
 ##
    # Logging Settings
    ##
    log_format json '{"method": "$request_method", "uri": 
 "$request_uri", "status": $status, "request_time": $request_time, 
 "upstream_response_time": $upstream_response_time, "body_bytes_sent": 
 $body_bytes_sent, "http_user_agent": "$http_user_agent", "remote": 
"$remote_addr", "upstream": "$upstream_addr", "proxy": "$proxy_host"}';
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;
    ##
    # Gzip Settings
    ##
    gzip on;
    gzip_disable "msie6";
    gzip_vary on;
gzip_proxied any;
    gzip_comp_level 9;
    #gzip_buffers 16 8k;
    #gzip_http_version 1.1;
    #gzip_types text/plain text/css application/json;
    gzip_types text/plain text/css application/json application/x-
javascript text/xml application/xml application/xml+rss text/javascript;
    ##
    # nginx-naxsi config
    ##
    # Uncomment it if you installed nginx-naxsi
    ##
    #include /etc/nginx/naxsi_core.rules;
    ##
    # nginx-passenger config
    ##
    # Uncomment it if you installed nginx-passenger
    ##
 include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

user_2451627 · Answer

First, identify which of server returns timeout error. Load balancer or Nginx Worker

user_9956279 · Answer

Hope this helps some people with similar setup.

Setup: IN MY CASE, On GCP, a NGINX instance was placed inside a subnet and for this subnet, a NAT Gateway was attached for the NGINX to access the public internet.

Since this NGINX used proxy_pass to a domain associated with our backend API, it needed access to the internet.

Errors encountered: I was getting 504 timeout on NGINX error log after multiple requests at the same time. In summary: 110 timeout while connecting to upstream

Solution: increase the minimum number of ports per VM instance on NAT Gateway.

No NGINX config changes were needed.

The minimum number of ports per VM instance was initially set to 64 which was not sufficient when the number of requests increased.

GCP Cloud NAT Overview

Nginx gateway timeout without much load on server

2 Answers