We're running an spring boot (2.3.0) application on a tomcat server (9.0.24) and every so often we get an Too many open files exception, and the application stops working.
I checked the limits of the tomcat instance:
Max open files 4096 4096 files
Running lsof for the process returns a long list of sock files all looking like this:
java 17063 tomcat 4085u sock 0,9 0t0 113306 protocol: TCP
root@popdb-dev:~# lsof -p 17063 | grep sock | wc -l
4003
However when I run ss to see what the sockets do I get a whole lot of closed sockets:
root@popdb-dev:~# ss -s
Total: 4384 (kernel 4485)
TCP: 4124 (estab 87, closed 4015, orphaned 0, synrecv 0, timewait 0/0), ports 0
Transport Total IP IPv6
* 4485 - -
RAW 1 0 1
UDP 10 6 4
TCP 109 54 55
INET 120 60 60
FRAG 0 0 0
As you can see most of the sockets are already closed. I searched stackoverflow and google but couldn't find any reason why the closed sockets are not cleaned up. When I stop or restart tomcat the closed sessions are cleaned up.
What can we do so the sockets are cleaned up and our application doesn't stop working?