Socket accept - "Too many open files"

There are multiple places where Linux can have limits on the number of file descriptors you are allowed to open.

You can check the following:

cat /proc/sys/fs/file-max

That will give you the system wide limits of file descriptors.

On the shell level, this will tell you your personal limit:

ulimit -n

This can be changed in /etc/security/limits.conf - it's the nofile param.

However, if you're closing your sockets correctly, you shouldn't receive this unless you're opening a lot of simulataneous connections. It sounds like something is preventing your sockets from being closed appropriately. I would verify that they are being handled properly.

On MacOS, show the limits:

launchctl limit maxfiles

Result like: maxfiles 256 1000

If the numbers (soft limit & hard limit) are too low, you have to set upper:

sudo launchctl limit maxfiles 65536 200000

it can take a bit of time before a closed socket is really freed up

lsof to list open files

cat /proc/sys/fs/file-max to see if there's a system limit

For future reference, I ran into a similar problem; I was creating too many file descriptors (FDs) by creating too many files and sockets (on Unix OSs, everything is a FD). My solution was to increase FDs at runtime with setrlimit().

First I got the FD limits, with the following code:

// This goes somewhere in your code
struct rlimit rlim;

if (getrlimit(RLIMIT_NOFILE, &rlim) == 0) {
    std::cout << "Soft limit: " << rlim.rlim_cur << std::endl;
    std::cout << "Hard limit: " << rlim.rlim_max << std::endl;
} else {
    std::cout << "Unable to get file descriptor limits" << std::endl;
}

After running getrlimit(), I could confirm that on my system, the soft limit is 256 FDs, and the hard limit is infinite FDs (this is different depending on your distro and specs). Since I was creating > 300 FDs between files and sockets, my code was crashing.

In my case I couldn't decrease the number of FDs, so I decided to increase the FD soft limit instead, with this code:

// This goes somewhere in your code
struct rlimit rlim;

rlim.rlim_cur = NEW_SOFT_LIMIT;
rlim.rlim_max = NEW_HARD_LIMIT;

if (setrlimit(RLIMIT_NOFILE, &rlim) == -1) {
    std::cout << "Unable to set file descriptor limits" << std::endl;
}

Note that you can also get the number of FDs that you are using, and the source of these FDs, with this code.

Also you can find more information on gettrlimit() and setrlimit() here and here.

Similar issue on Ubuntu 18 on vsphere. The cause - Config file nginx.conf contains too many log files and sockets. Sockets are treated as files in Linux. When nginx -s reload or sudo service nginx start/restart, the Too many open files error appeared in error.log.

NGINX worker processes were launched by NGINX user. Ulimit (soft and hard) for nginx user was 65536. The ulimit and setting limits.conf did not work.

The rlimit setting in nginx.conf did not help either: worker_rlimit_nofile 65536;

The solution that worked was:

$ mkdir -p /etc/systemd/system/nginx.service.d
$ nano /etc/systemd/system/nginx.service.d/nginx.conf
    [Service]
    LimitNOFILE=30000
$ systemctl daemon-reload
$ systemctl restart nginx.service

When your program has more open descriptors than the open files ulimit (ulimit -a will list this), the kernel will refuse to open any more file descriptors. Make sure you don't have any file descriptor leaks - for example, by running it for a while, then stopping and seeing if any extra fds are still open when it's idle - and if it's still a problem, change the nofile ulimit for your user in /etc/security/limits.conf