I have spring-boot (version: 2.2.1.Release) application. The application has a scheduled task (call it Task-A) that makes a lot of requests to third api, that occasionally may be down. Also the application has an opened WebSocket, so that client may check in real-time status of some process. Web socket has following configuration:
@Configuration
@EnableWebSocketMessageBroker
@EnableWebSocket
class WebSocketConfig : WebSocketMessageBrokerConfigurer {
override fun registerStompEndpoints(registry: StompEndpointRegistry) {
registry.addEndpoint("/ws/activity")
.setAllowedOrigins("origin-from-where-connection-to-socket-comes.com")
.withSockJS()
}
}
And there is the second scheduled task (call it Task-B) that writes information for clients into the socket every 5 seconds:
@Component
class ChargersScheduled
@Autowired constructor(
private val processMonitor: ProcessMonitor,
private val messagingTemplate: SimpMessagingTemplate
) {
@Scheduled(fixedDelay = 5000)
fun getSchedulersActivity () {
messagingTemplate.convertAndSend("/web-socket/activity", processMonitor.checkActivity())
}
}
At some point of time Task-A starts to emit IOException : Too many open files, a minute or two later logs are starting to fill with:
o.a.t.u.n.Acceptor : Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:461)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:73)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.lang.Thread.run(Thread.java:748)
Is that caused by clients attempting to connect to WebSockets or making simple requests to the server or, perhaps, both? Aside from increasing ulimit (already done) what is the way of mitigating the problem? As for now I had to restart the application since it hanged, as if it was under DDOS attack.