502 Bad Gateway with DigitalOcean (gunicorn/nginx) using Selenium and Django

This error message...

502 Bad Gateway
nginx/1.14.0 (Ubuntu)

...using DigitalOcean (gunicorn/nginx) can occur for a number of reasons. Determining the exact cause of 502 error varies depending on the webserver you are using as well as the application server interpreting the requests.

502 Bad Gateway

Bad gateway errors are typically caused by a breakdown in communication between webserver and application handler. In many cases, the underlying issue is either excessive latency or exceedingly short timeout windows.

At times, 502 Bad Gateway is also caused by a misconfigured application server, when the webserver understood the request and passed it along to the appropriate handler, but a disruption of some sort occurred between the two.

Solution

Gunicorn being one of the widely used Python WSGI server, diagnosing the cause of 502 Bad Gateway error would mostly depend on the application server you are using and some of the common issues are as follows:

• Gunicorn is not running: You need to ensure that Gunicorn is running using ps. To ensure Gunicorn is running you must see a similar output:

  www-data@nginx0:/var/log/nginx$ ps aux | grep gunicorn
  www-data     13805  0.0  1.8  52292 18460 pts/0    S    20:32   0:00 /home/www-data/test_app/bin/python /home/www-data/test_app/bin/gunicorn --error-logfile /var/log/gunicorn/errors.log -b 0.0.0.0:8080 wsgi
  www-data     13836  0.0  1.5  52432 15392 pts/0    S    20:34   0:00 /home/www-data/test_app/bin/python /home/www-data/test_app/bin/gunicorn --error-logfile /var/log/gunicorn/errors.log -b 0.0.0.0:8080 wsgi
  

• Gunicorn won’t start: At times, Gunicorn won’t start due to a typo in the configuration file or port conflict or inaccessible log directory or any combination of these situations. In these cases to check your Gunicorn configuration execute the command:

  gunicorn --check-config [APP_MODULE]
  

• NGINX is misconfigured: Incase Gunicorn is properly configured, it may be possible that NGINX is not looking for it in the right place. In those cases, open your NGINX configuration file (/etc/nginx/nginx.conf) and look for a block starting with the upstream keyword as follows:

  upstream app_servers {
      server 127.0.0.1:8080;
  }
  

  This setting is to configure NGINX to redirect requests for _app_servers_ to, in this case, 127.0.0.1:8080. If Gunicorn is not bound to 127.0.0.1 or is not listening on 8080, either change Gunicorn’s configuration to match NGINX’s, or change NGINX’s configuration to match Gunicorn’s. Additionally, verify that your site configuration is redirecting your application to the appropriate upstream server. To ensure that, you need to open your site’s configuration i.e. /etc/nginx/sites-enabled/your_site and look for the block defining the URL endpoint to your application. As an example:

  location /my-app {
          proxy_pass         http://app_servers;
          proxy_redirect     off;
          proxy_set_header   Host $host;
          proxy_set_header   X-Real-IP $remote_addr;
          proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header   X-Forwarded-Host $server_name;
  
  }
  

• Gunicorn is timing out: Incase your application takes a long time to respond ( > 30s by default), Gunicorn may return a 502 to NGINX. This can be verified by checking the Gunicorn logs (defaults to STDOUT if no logfile set). As an example,

  [2016-09-21 20:33:04 +0000] [13805] [CRITICAL] WORKER TIMEOUT (pid:13810)
  

  The above log implies that, the application is taking too long to respond to Gunicorn, resulting in the worker thread being killed because Gunicorn thinks the worker hung. In such cases, increasing Gunicorn’s max execution time would be the best solution. However, from the application and the data set processing perspective increasing timeout windows may not be the best solution and you may require to profiling and optimizing the application in use.

• Tweaking read_timeout: If you still see 502 Bad Gateway after modifying Gunicorn’s timeout threshold, you need to follow these steps mentioned below to increase the timeout window for NGINX:

  • Open your NGINX config (/etc/nginx/nginx.conf)
  • Add fastcgi_read_timeout XXX; within the http block, where XXX is the timeout window in seconds (see below for an example)
  • Save and close the file
  • Reload your NGINX config

  • An example:

    http { 
    ...
    
    fastcgi_buffers 8 16k;
    fastcgi_buffer_size 32k;
    fastcgi_connect_timeout 300;
    fastcgi_send_timeout 300;
    fastcgi_read_timeout 300;
    }
    

1/502 Bad Gateway caused by GUNICORN
a - sudo nano /etc/systemd/system/gunicorn.service.
b-

[Unit]
Description=gunicorn daemon
After=network.target

[Service]
User=sammy
Group=www-data
WorkingDirectory=/home/sammy/myproject
ExecStart=/home/sammy/myproject/myprojectenv/bin/gunicorn --access-logfile - --timeout 300 --workers 3 --bind unix:/home/sammy/myproject/myproject.sock myproject.wsgi:application

[Install]
WantedBy=multi-user.target

c -

sudo systemctl start gunicorn
sudo systemctl enable gunicorn
sudo systemctl daemon-reload
sudo systemctl restart gunicorn

2/504 Bad Gateway caused by NGINX
a - sudo nano /etc/nginx/nginx.conf
b - add those to http

client_body_timeout 999;
client_header_timeout 999;
keepalive_timeout 999;
fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
fastcgi_connect_timeout 999;
fastcgi_send_timeout 999;
fastcgi_read_timeout 999;

c - change this in events

worker_connections 1024;

d - service nginx reload