Backstory: We manage infrastructure for many customers and almost all have Nginx proxy where they terminate SSL and do some caching. We have never encountered any problems on Nginx and we hit a wall during this troubleshooting.
Problem: Recently we did a migration for new customer and their application uses socket.io. We noticed that memory usage is only rising as time goes by and only one worker process is causing that. Restarts do not help, when 1 worker is getting "locked", RAM usage only increases. Everything works well and nothing is slow, but it could cause OOM and that would be a problem. When a worker is killed, RAM is released but then the new one does the same thing.
Setup: 4vCPU/8GB RAM Nginx VM (version 1.18.0) running on ESXi hypervisor - Ubuntu 18.04.4 LTS. Nginx is configured only as a proxy, serving around 150-200 req/sec. No caching, no serving static files. We examined pmap and we saw that this is considered as dirty memory, but we don't know what is causing that. No errors in syslog or nginx error.log. Access logs are turned off and if you want to do config reload the worker that has problems hangs in "worker process is shutting down" state. We assume something is not properly configured for socket.io on Nginx side, but it's hard to figure out what.
Thanks a lot for help and apologise for grammar. I can provide more info/metrics/config if needed.
Some config parameters:
events {
worker_connections 4096;
use epoll;
multi_accept on;
}
http {
aio threads;
keepalive_timeout 65;
proxy_buffering on;
proxy_read_timeout 120s;
proxy_send_timeout 120s;
proxy_buffers 8 2m;
proxy_buffer_size 12m;
proxy_busy_buffers_size 12m;
proxy_headers_hash_max_size 1024;
proxy_headers_hash_bucket_size 128;
....
}