I have a Spring Boot WebFlux service that is I/O intensive and calls external resources.
Usually, all works fine. But from time to time, I noticed the service looks like it was freezing/hibernating. This means that it can be idle for minutes between two lines of code that normally takes a few milliseconds to pass.
This is usually around I/O operations. For example, getting a response from resource A and then calling resource B.
We have noticed sometimes 10 minutes between these - as if the service has no available thread to work on the request or it is queueing somewhere.
This freeze may be related to slow responses from the external service we are calling. We were expecting that WebFlux will handle it with no issues, but it seems like the external resource's slowness is affecting our service.
Pseudo normal scenario:
10:00:00 - call external service A (responses within 2 seconds)
10:00:02 - getting a response from service A
10:00:02:10 - calling service B after 10 Milliseconds
Pseudo malfunctioned scenario:
10:00 - call external service A (responses are getting slower, say to 1 minute)
10:01 - getting a response from service A
10:11 - calling service B <----- why did it wait 10 minutes?
When this happens:
- The number of requests to the service is moderate and constant (~40 per second)
- The CPU is low (~20-40%) and not spiking.
- Number of threads is increasing from ~100 to ~150.
Any ideas what can cause such a thing?