Jetty IOException: Too many open files

Viewed 16500

I'm running Jetty on a website doing around 100 requests/sec, with nginx in front. I just noticed in the logs, only a few minutes after doing a deploy and starting Jetty, that for a little while it was spamming:

java.io.IOException: Too many open files
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
    at org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:75)
    at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:673)
    at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

For a minute or two. I did an "lsof -u jetty" and saw hundreds of lines of:

java    15892 jetty 1020u  IPv6          298105434        0t0       TCP 192.168.1.100:http-alt->192.168.1.100:60839 (ESTABLISHED)
java    15892 jetty 1021u  IPv6          298105438        0t0       TCP 192.168.1.100:http-alt->192.168.1.100:60841 (ESTABLISHED)
java    15892 jetty 1022u  IPv6          298105441        0t0       TCP 192.168.1.100:http-alt->192.168.1.100:60842 (ESTABLISHED)
java    15892 jetty 1023u  IPv6          298105443        0t0       TCP 192.168.1.100:http-alt->192.168.1.100:60843 (ESTABLISHED)

Where 192.168.1.100 is the servers internal IP.

As you can see, this brought the number of open files to the default max of 1024. I could just increase this, but I'm wondering why this happens in the first place? It's in Jetty's nio socket acceptor, so is this caused by a storm of connection requests?

2 Answers

I solved similar issue which was caused due to a bug in Spring Boot application (versions 2.2.8 or 2.3.1) by upgrading Spring boot to latest 2.4.1

Related