OOM killed JVM with 320 x 16MB Netty DirectByteBuffer objects

user_1672799 0
Viewed 2603

I run an app in a 7.5GB RAM server (no swap) with args:
-Xmx3g -Xms3g -Xlog:gc -XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -XX:MaxDirectMemorySize=500m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=256m -Dio.netty.maxDirectMemory=0 -Djdk.nio.maxCachedBufferSize=104857

The RSS of the process keeps increasing while the heap is 90% empty until the linux memory killer takes action.

Took a heap dump:
SELECT db, db.capacity, classof(inbounds(db).get(0)).getName() FROM java.nio.DirectByteBuffer db WHERE classof(inbounds(db).get(0)).getName().startsWith("io.netty")

And it results on 320 instances of DirectByteBuffer with capacity==16MB from io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry

(I guess not all the "capacity" is physically allocated because it would exceed 5GB)

The app is not using Netty directly but some dependencies are: S3, Redisson, version -> 4.1.58. Google cloud monitoring with a grpc-netty-shaded (1.34.1)

How I can control the amount of memory used?

netty java
2 Answers

Netty has several java properties that you can adjust according to your needs.

By default, Netty use Direct memory allocation in a buffer pool allocator. Pool allocator is highly recommended since it reuse memory, whatever direct or not. I recommend to not change the default Buffer Pool Allocator.

So here are some properties to set at your Java command startup:

  • -Dio.netty.noPreferDirect=true (by default is false): if true, will not use Direct Memory from pool allocator but Heap Memory ; note it might slow down a bit since Direct Memory is fastest and you need to allocate more Memory to JVM using -Xms -Xmx

  • -Dio.netty.maxDirectMemory=x (by default is -1): you can change this value

    • If <0, means Netty use twice Direct memory from JDK and don't use cleaner. default value
    • If 0, means Netty use the max JDK Direct ram and use Cleaner (recommended).
    • If >0, means Netty use this max Direct, not related to max of JDK, and don't use cleaner.

So for instance, if you can afford (and it is preferable) to use Direct Memory for Netty, but no more than 1 GB, you could set:

  • -Dio.netty.maxDirectMemory=0

If you don't want any Direct Buffer, then you could set:

  • -Dio.netty.noPreferDirect=true
  • -Dio.netty.maxDirectMemory=0

End but not least, in addition to one of the previous settings:

  • -XX:MaxDirectMemorySize=500m might be too small if using Direct Memory, maybe increase this as -XX:MaxDirectMemorySize=1024m or more
  • you may decrease -Xmx3g -Xms3g such as -Xmx2g -Xms2g
user_42321320

You cannot control the amount of memory, if not causing OOM as you have done. Netty pooling won't behave like the Java GC vs heap ie increasing some throttling/frequency of its work in order to use resources within specified limits (throwing OOM just under specific circumstances). Netty memory pooling is built to mimic the behaviour of a native allocator eg jemalloc, hence its purpose is to retain as much memory as the application need to work. For this reason, the retained direct memory depends by the allocation pressure that the application code perform ie how many outstanding alloc without release.

I suggest, instead, to embrace its nature, prepare an interesting test load on a preprod/test machine and just monitor the Netty direct memory usage of the application you're interested in. I suppose you've configured -Dio.netty.maxDirectMemory=0 for the purpose of using JMX to expose the direct memory used, but Netty can expose it's own metrics as well (saving setting io.netty.maxDirectMemory), just check that the libraries that use it take care of exposing through JMX or using whatever metrics framework. If these applications won't expose it, the API is fairly easy to be used, see https://netty.io/4.1/api/io/netty/buffer/PooledByteBufAllocatorMetric.html

user_63154440
Related