We are having a memory leak in a high load(6m req/day) Spring boot application, serving few rest endpoints. old_gen grow slowly over time and committed memory reaches the container limit, causing container showdown. After taking several heapdump, I was able to find out the following:
A good amount of memory is used by org.apache.catalina.webresources.CachedResource, with an instance count of around 12000, causing similar amount of other objects including java.io.File. I tried disabling tomcat caching, while these classes are gone from heapdump but the throughput decreases inevitably. Any suggestions on the best practice?
Spring actuator also consumes good amount of memory
Class Name | Objects | Shallow Heap | Retained Heap
--------------------------------------------------------------------------------------------------------------------------------
org.springframework.boot.actuate.audit.InMemoryAuditEventRepository | 1 | 24 | >= 5,670,928
org.springframework.boot.actuate.audit.AuditEvent[] | 1 | 4,016 | >= 5,670,880
org.springframework.boot.actuate.audit.AuditEvent | 1,000 | 32,000 | >= 5,666,416
org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationDetails| 1,000 | 40,000 | >= 5,319,552
--------------------------------------------------------------------------------------------------------------------------------It seems to be stable after reaching this level. But in that case on every request, the objects are being replaced in the array. The code(InMemoryAuditEventRepository.java) looks like below, not sure - maybe this causes some leak.
this.events[this.tail] = event;
Any suggestions on this? disable spring actuator?
- 17670 instances of this class: org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource$DefaultCacheKey. Also seems stable like previous class. Seems like related to caching.
- 20000 instances of com.mysql.jdbc.ConnectionPropertiesImpl$BooleanConnectionProperty!
Path to GC shows:
<pre>
Class Name | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller @ 0xcadf60f8 NioBlockingSelector.BlockPoller-1 Thread| 16,884 | 136 | 1,080,576 | 920
'- contextClassLoader org.springframework.boot.loader.LaunchedURLClassLoader @ 0xc8018578 | 16,884 | 80 | 1,080,576 | 4,333,096
'- classes java.util.Vector @ 0xc82b5c90 | 16,884 | 32 | 1,080,576 | 2,706,712
'- elementData java.lang.Object[20480] @ 0xca1a5dd0 | 16,884 | 81,936 | 1,080,576 | 2,706,680
'- [7996] class com.mysql.jdbc.NonRegisteringDriver @ 0xc9b94bd8 | 16,884 | 112 | 1,080,576 | 7,272
'- connectionPhantomRefs java.util.concurrent.ConcurrentHashMap @ 0xc9b94ed8 | 16,884 | 64 | 1,080,576 | 6,528
'- table java.util.concurrent.ConcurrentHashMap$Node[256] @ 0xcc905b78 | 16,884 | 1,040 | 1,080,576 | 6,464
|- [15] java.util.concurrent.ConcurrentHashMap$Node @ 0xcc963078 | 504 | 32 | 32,256 | 128
| '- val, key com.mysql.jdbc.NonRegisteringDriver$ConnectionPhantomReference @ 0xcbfa3840 | 504 | 32 | 32,256 | 48,504
| |- discovered com.mysql.jdbc.NonRegisteringDriver$ConnectionPhantomReference @ 0xcbb4ec50| 378 | 32 | 24,192 | 48,504
| |- referent com.mysql.jdbc.JDBC4Connection @ 0xcc080018 | 126 | 1,232 | 8,064 | 32,024
| '- Total: 2 entries | | | |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Any clue?
- Heapdump also shows some unreachable objects. Count 128,249 consuming 7MB, major of which are:
<pre>
Class Name | Objects | Shallow Heap
---------------------------------------------------------------------------
char[] | 21,936 | 1,988,712
int[] | 13,191 | 1,913,760
java.lang.Object[] | 12,230 | 652,176
java.lang.String | 21,936 | 526,464
java.lang.reflect.Method | 2,994 | 263,472
java.lang.ref.SoftReference | 4,051 | 162,040
java.util.LinkedHashMap$Entry | 3,873 | 154,920
java.util.HashMap$Node[] | 2,470 | 142,784
java.util.LinkedHashMap | 2,314 | 129,584
java.lang.reflect.Constructor | 1,133 | 90,640
sun.reflect.generics.tree.SimpleClassTypeSignature| 2,776 | 66,624
java.util.ArrayList | 2,735 | 65,640
---------------------------------------------------------------------------Can anyone suggests how to see the value of these objects (using MAT), how to track down?
Any other suggestions?