Linux - troubleshooting the "too many open files" error

user_5224236 0
Viewed 960

On RHEL, I have the "too many open files" error. I'd like to reproduce it so that I understand what exactly is causing it.

  • cat /proc/sys/fs/file-max gives the max open files at OS-level. ./file-nr gives the number of current open files
  • sudo lsof | wc -l is another way to look at the amount of current open files alhough it contains duplicated values as mentioned here.
  • ulimit -n is the max open files per process (for me 1024)
  • sudo lsof -p <PID> | wc -l gives the lsof version of the number of open files for a current process.

I would be after a bash solution if possible, that creates a process that blows up the ulimit -n just by opening dummy files. Potentially, I'd like to use several of those test processes to blow up the OS limit as well.

Also how can I properly kill such processes? Thanks

This will allow me to check exactly what limit is reached (file-max or lsof...) and to experiment on what kind of operations are causing it.

linux bash
3 Answers
#!/usr/bin/env bash
case $BASH_VERSION in
  ''|[1-3].*|4.0.*) echo "ERROR: Bash 4.1 or newer required" >&2; exit 1;;
esac

[[ $1 ]] || { echo "Usage: $0 number-of-files" >&2; exit 1; }

echo "Running as process $$" >&2
for ((i=0; i<$1; i++)); do
  exec {fd_num}>/dev/null || {
    echo "Error on iteration $((i+1))" >&2
    break
  }
done

# if started from a TTY, let user press enter to exit
# otherwise (f/e, if run with </dev/null), keep running until we're killed
if [[ -t 0 ]]; then
  read -p "Press enter to exit this process:" _
else
  while :; do sleep 600 || exit; done
fi

If this outputs Running as process 26941, you can then ls -l /proc/26941/fd to see that the file descriptors are genuinely in use (and count them, if you choose).

user_141220

You could create 2000 file descriptors to test the ulimit, if the number set is lower than 2000.

Give a try to this:

inc=1
while [ $inc -lt 2000 ] ; do
  exec {file_descriptor}>/tmp/dummy_file_$inc.txt
  inc=$((inc+1))
done
user_60103430

If you don't mind a simple Perl solution:

#!/usr/bin/env perl
# Open maximum number per session file descriptors
use strict;
use warnings;
my $secs = shift || 10;
my @fh;
while (1) {
    my $fh;
    open( $fh, ">", "/dev/null" ) or last;
    push( @fh, $fh );
}
print $!, " [ ", scalar @fh, " ]\n";
sleep $secs;
1;

The script can be run with one optional argument denoting the time to sleep after the ulimit exhaustion. The default wait time is 10 seconds.

On a system with a ulimit of 1024:

Too many open files [ 1021 ]

Upon exhaustion of the ulimit, the operating system reason for the open failure is returned noting the number of descriptors created. As every process has three standard file descriptors, the limit is reached is ulimit -3.

user_10153850
Related