One of our PRD small win-service suddenly skyrocketed to 80+ GB memory for a few hours and then came down and stabilized at 6+ GB memory, it is usually use only under 200 MB. I have grab a memory dump at the time of 6+ GB, and restarted the server, everything go back to normal. I am investigating why it it skyrocketed to that high and why it's stabled at 6+ GB.
I have found that it is an unmanaged memory leak issue, because the CLR heap is small:
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x000000910083ff20
generation 1 starts at 0x00000091007caba0
generation 2 starts at 0x0000009100001000
ephemeral segment allocation context: none
segment begin allocated size
0000009100000000 0000009100001000 00000091029691f0 0x29681f0(43418096)
Large object heap starts at 0x0000009110001000
segment begin allocated size
0000009110000000 0000009110001000 000000911013dcb8 0x13ccb8(1297592)
Total Size: Size: 0x2aa4ea8 (44715688) bytes.
------------------------------
***GC Heap Size: Size: 0x2aa4ea8 (44715688) bytes.***
But the Heap Summary is high, More suspiciously, the lock count is high:
0:000> !heap -s
************************************************************************************************************************
NT HEAP STATS BELOW
************************************************************************************************************************
LFH Key : 0x45d65d36d8f8b642
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
**000000917aa90000 00000002 6718396 6682116 6718196 71860 5580 419 2 93 LFH**
000000917a850000 00008000 64 4 64 2 1 1 0 0
000000917acd0000 00001002 1280 88 1080 15 7 2 0 0 LFH
000000917ac90000 00001002 1280 108 1080 24 8 2 0 0 LFH
000000917b160000 00001002 1280 124 1080 7 10 2 0 0 LFH
000000917b310000 00041002 60 8 60 5 1 1 0 0
000000917bd40000 00041002 260 36 60 4 3 1 0 0 LFH
0000009118080000 00001002 1084 1024 1084 1021 2 1 0 0
-------------------------------------------------------------------------------------
0:000> !heap -stat -h 000000917aa90000
heap @ 000000917aa90000
group-by: TOTSIZE max-display: 20
size #blocks total ( %) (percent of total busy bytes)
2d0b0 373e - 9b845aa0 (39.37)
10d1 41962 - 44eed902 (17.45)
ddc8 373e - 2fdbae70 (12.12)
7ab8 373e - 1a7b4090 (6.70)
7168 373e - 1878cf30 (6.20)
29d1 6e6b - 1209485b (4.57)
35d1 372d - b995cbd (2.94)
31d1 372d - abca8bd (2.72)
12d1 6e5a - 81c6b7a (2.05)
21d1 3746 - 74d2626 (1.85)
fd1 6e6c - 6d27a2c (1.73)
1cd1 372d - 635f7bd (1.57)
40 22b52 - 8ad480 (0.14)
100 6ef7 - 6ef700 (0.11)
200 374f - 6e9e00 (0.11)
c3500 5 - 3d0900 (0.06)
80 6edb - 376d80 (0.05)
d8 374e - 2ea9d0 (0.05)
249f18 1 - 249f18 (0.04)
90 3792 - 1f4220 (0.03)
Then I did !heap -flt s 2d0b0
I can't do !heap -p -a to see the stack because I don't have gflags enabled in PRD.
Instead, I am trying to do a dc 00000092b7088bb0 L200 to guess base on the string value. Nothing meaningful in the first chuck (2d0b0 373e - 9b845aa0 (39.37) But in the second chuck ( 10d1 41962 - 44eed902 (17.45)), I found something like
00000092`b70b0250 00000000 00000000 44440000 4e4f4d2d ..........DD-MON
00000092`b70b0260 0052522d 00000000 00000000 00000000 -RR.............
00000092`b70b0270 00000000 00000000 00000000 00000000 ................
00000092`b70b0280 00000000 00000000 00000000 00000000 ................
00000092`b70b0290 00000000 48480000 2e494d2e 46585353 ......HH.MI.SSXF
00000092`b70b02a0 4d412046 00000000 00000000 00000000 F AM............
00000092`b70b02b0 00000000 00000000 00000000 00000000 ................
00000092`b70b02c0 00000000 44440000 4e4f4d2d 2052522d ......DD-MON-RR
00000092`b70b02d0 4d2e4848 53532e49 20464658 00004d41 HH.MI.SSXFF AM..
00000092`b70b02e0 00000000 00000000 00000000 00000000 ................
00000092`b70b02f0 00000000 00000000 00000000 00000000 ................
00000092`b70b0300 00000000 00000000 00000000 00000000 ................
00000092`b70b0310 00000000 48480000 2e494d2e 46585353 ......HH.MI.SSXF
00000092`b70b0320 4d412046 525a5420 00000000 00000000 F AM TZR........
00000092`b70b0330 00000000 00000000 00000000 00000000 ................
00000092`b70b0340 00000000 00000000 00000000 00000000 ................
00000092`b70b0350 00000000 00000000 44440000 4e4f4d2d ..........DD-MON
00000092`b70b0360 2052522d 4d2e4848 53532e49 20464658 -RR HH.MI.SSXFF
00000092`b70b0370 54204d41 0000525a 00000000 00000000 AM TZR..........
It looks like Oracle Database Query Date Time Format string. And We do use Oracle, and I am hard to believe that we are leaking oracle connections, because this small service have run for years and this is the first time this happened. That's how far I can go now.
Sorry for the long question and thanks for spend time reading
To summarize my question:
- what is the Lock Count Column mean in !heap -s , that looks like an indicator of an issue where I can keep digging, please point me a direction or blog I can read.
- Are those Raw data string ring another bell on you instead of Oracle? Have you ever have similar issue?
- We do use a lot .net remoting. I know it's old and I never deep dived into it. Does it could be a factor of this issue?
- I can't do anything in PRD, and I can't reproduce this issue in DEV/TEST, is there any other direction I can take?