I have an ASP.NET Core application running on Linux (in Docker, on Kubernetes). I am running into the situation where my fully managed application uses 5 MB on the GC heap, but the application reserves 5 GB (and will consume more until it is killed via OOM). I collected a memory dump using dotMemory and I can see that the memory is consumed by unmanaged memory. I get the same picture by calling GC.GetTotalMemory vs Process.GetCurrentProcess().WorkingSet64.
The code is fully managed mostly reading files, processing them, and storing the processed data in a database. I use DevArt.PostgreSql but also tried Npgsql and it makes no difference. I also use a single raw, persistent Socket connection. In the given test case, there are no calls to Web API on ASP.NET Core. dotMemory does not show any managed leaks in those library, nor in my part of the code.
Memory Usage will look like this:
Process.GetCurrentProcess().WorkingSet64=582 MB / GC.GetTotalMemory=372 MB
and after a while jump to
Process.GetCurrentProcess().WorkingSet64=600 MB / GC.GetTotalMemory=5 MB
I can force this collection using the following code:
GC.Collect(2, GCCollectionMode.Forced, true, true);
GC.WaitForPendingFinalizers();
GC.Collect(2, GCCollectionMode.Forced, true, true);
So, I can conclude more than 300 MB were transferred from GetTotalMemory to WorkingSet64. I don't see how else to arrive at those numbers. When memory fills up again, it can look like this:
Process.GetCurrentProcess().WorkingSet64=1375 MB / GC.GetTotalMemory=4 MB
(few moments later)
Process.GetCurrentProcess().WorkingSet64=1433 MB / GC.GetTotalMemory=1081 MB
In this case, a lot of GC memory could be reclaimed from the WorkingSet. However, this cycle is never perfect and the increase in WorkingSet64 goes up until 50+ GB and the process is killed due to OOM.
The machine has high Core Count (32) and RAM (64 GB). I tried both variants of GC, but made no difference.
Somehow, the memory is just not released back to the OS.
Here's my LOH
Here's my 2nd generation
Gen 0 and 1 are empty.
Any suggestions how I might find the root cause would be appreciated.