Nginx and php-fpm: cannot get rid of 502 and 504 errors

Question

I have an ubuntu-server and a pretty high loaded website. Server is:

Dedicated to nginx, uses php-fpm (no apache), mysql is located on different machine
Has 8 GB of RAM
Gets about 2000 requests per second.

Each php-fpm process consumes about 65MB of RAM, according to top command:

top command

Free memory:

admin@myserver:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          7910       7156        753          0        284       2502
-/+ buffers/cache:       4369       3540
Swap:         8099          0       8099

PROBLEM

Lately, I'm experiencing big performance problems. Very big response times, very many Gateway Timeouts and in evenings, when load gets high, 90% of the users just see "Server not found" instead of the website (I cannot seem to reproduce this)

LOGS

My Nginx error log is full of the fallowing messages:

2012/07/18 20:36:48 [error] 3451#0: *241904 upstream prematurely closed connection while reading response header from upstream, client: 178.49.30.245, server: example.net, request: request: "GET /readarticle/121430 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9001", host: "example.net", referrer: "http://example.net/articles"

I've tried switching to unix socket, but still get those errors:

2012/07/18 19:27:30 [crit] 2275#0: *12334 connect() to unix:/tmp/fastcgi.sock failed (2: No such file or directory) while connecting to upstream, client: 84.
237.189.45, server: example.net, request: "GET /readarticle/121430 HTTP/1.1", upstream: "fastcgi://unix:/tmp/fastcgi.sock:", host: "example.net", referrer: "http
://example.net/articles"

And php-fpm log is full of these:

[18-Jul-2012 19:23:34] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there  are 0 idle, and 75 total children

I've tried to increase given parameters up to 100, but it still seems not enough.

CONFIGS

Here is my current configuration

php-fpm

listen = 127.0.0.1:9001
listen.backlog = 4096
pm = dynamic
pm.max_children = 130
pm.start_servers = 40
pm.min_spare_servers = 10
pm.max_spare_servers = 40
pm.max_requests = 100

nginx

worker_processes  4;
worker_rlimit_nofile 8192;
worker_priority 0;
worker_cpu_affinity 0001 0010 0100 1000;

error_log  /var/log/nginx_errors.log;

events {
    multi_accept off;
    worker_connections  4096;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    access_log off;
    sendfile        on;
    keepalive_timeout  65;
    gzip  on;

    # fastcgi parameters
    fastcgi_connect_timeout 120;
    fastcgi_send_timeout 180;
    fastcgi_read_timeout 1000;
    fastcgi_buffer_size 128k;
    fastcgi_buffers 4 256k;
    fastcgi_busy_buffers_size 256k;
    fastcgi_temp_file_write_size 256k;
    fastcgi_intercept_errors on;

    client_max_body_size 128M;

    server {
        server_name example.net;
        root /var/www/example/httpdocs;
        index index.php;
        charset utf-8;
        error_log /var/www/example/nginx_error.log;

        error_page 502 504 = /gateway_timeout.html;

        # rewrite rule
        location / {
            if (!-e $request_filename) {
                rewrite ^(.*)$ /index.php?path=$1 last;
            }
        }
        location ~* \.php {
            fastcgi_pass 127.0.0.1:9001;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_param PATH_INFO $fastcgi_script_name;
            include fastcgi_params;
        }
    }
}

I would be very grateful for any advice on how to identify the problem and what parameters I can adjust to fix this. Or maybe 8GB of RAM is just not enough for this kind of load?

Nginx and php-fpm: cannot get rid of 502 and 504 errors

5 Answers