I have a Spring Boot application. It is a WebSocket server app, where I keep & manage active sessions in the various data structure, mostly ConcurrentHashMap and CopyOnWriteArrayList. Now I am doing load testing using Jmeter.
Using Jmeter, I am spinning up 1200 WS sessions within 300 seconds. And those session stays connected until I end test. To stress more, I add some more user sessions by running another Jmeter instance.
Here is what I observed the memory consumption for our service running on k8s.
- When I deploy/start the service, the memory consumption is around 165 MB
- Then I start Jmeter load test, which creates 1200 sessions in 300 seconds. This adds up to memory and it reached to around 400 MB, which is obvious.
- After a few minutes (20-30 minutes), I end the test, which brings down the no of sessions to zero. But the memory stays around 400 MB. I waited for almost 30 minutes but it does not decrease. And I had added logs to print the size of various data structure objects which hold the user session-related data. They all are zero. So technically I should see memory at the previous level (165 MB). But that's not happening until I restart the service/application.
To check the memory leak, I run the application locally and monitor via VisualVM. The behavior is the same. As you can see below screenshot, the spike in the threads indicates the time when I started the Jmeter load test. And the end of that spike indicates the time when I stopped the load test. But there is no difference in the heap space.
So why the GC does not reclaim memory? Am I missing anything? Because it just stays at the last peak value. For e.g. if I add 1500 user sessions in the load test and memory grows to 500 MB it stays at 500. I am afraid, what if with the time (days) the service running on k8s and will go out of memory.
Appreciate suggestions to improve here.