The support team in our company reported our API process is consuming too much RAM, so I started analyzing the issue, I've used visualvm and jconsole to monitor the process, and when I call an endpoint that has JNI calls to C++ native code, I see the resident memory (RES) in my process on the top linux monitoring software grow above the maximum heap size (-Xmx JVM flag). For example -Xmx3G but top shows 6G after hitting the endpoint many times.
Initially we thought it could be a native memory leak but I've run a test where I called the GC manually with System.gc() every five seconds and RES memory behaved properly.
With that result in mind, I started studying JVM flags and garbage collection. I've made a test with the serial collector and RES memory did well, but since it's an API I prefered using the G1GC collector. Calling the garbage collection more often without doing it explicitly seemed like an interesting fix. However, the solution I found is to set the new generation size to a low value, I've noticed that significant spike in RES memory was directly related to a spike in eden space analyzing the JConsole.
So i set the flag to -Xmn256M to try and control this RES memory spike that doesn't seem to be allocated inside normal Heap space, and to try and force garbage collection to be run more often, at least the collection from young to old gen. However, I've noticed G1GC is making many young colllections and no old collections.
As you can see young collections were called 47 times, but old ones were never called.
So, in summary: I've controlled memory spikes in resident memory by running young collections frequently with the G1GC because I've noticed that the garbage collection fixes this memory surplus (RES >> Max Heap Size), even though wasn't able to explain it, and even though I know this frequent young collection may affect application performace a little, because G1GC young generation is collected by stopping all application threads (from Java Performance by Scott Oaks). Is this a bad idea? Should I dedicate more time to diagnosing a memory leak (Java or native C++)?
My time with this task is almost up