Hopefully I've got all the messy details right here, it's a bit subtle.
By default Netty will usually set io.netty.maxDirectMemory=MaxDirectMemorySize and enable "no cleaner" buffers if possible. If "no cleaner" buffers are in use, Netty's direct memory & Java's "native" direct memory are tracked independently -- because Netty needs to do its own accounting to track memory allocated by Unsafe.allocateMemory in "no cleaner" buffers.
This means that the theoretical maximum memory set aside for off-heap data in a Netty service running default config is often roughly io.netty.maxDirectMemory+MaxDirectMemorySize -- or 2 x MaxDirectMemorySize by default. This is kind of quietly documented over here:
So that's fine, even if it is a bit surprising.
But things seem to get a little weird when you try to explicitly set io.netty.maxDirectMemory and MaxDirectSize to different values. For example, we were trying to clamp down our theoretical memory ceiling a little to play nice with cgroups: the service in question was getting OOM killed, so we got a little aggressive and, once we realized the relationship between MDMS/i.n.mdm, set MaxDirectMemorySize and io.netty.maxDirectMemory explicitly to minimize our ceiling -- -XX:MaxDirectMemorySize=1g and set -Dio.netty.maxDirectMemory=3221225472 with "no cleaner" direct buffers enabled.
I expected this effectively be a no-op change as far as Netty was concerned since we had -XX:MaxDirectMemorySize=3g before we explicitly set io.netty.maxDirectMemory. However, we observed drastically less in-use direct memory reported. Suspecting it probably had something to do with the reduced MaxDirectMemory size I eventually found this, which I think could explain the difference.
Why use PlatformDependent.maxDirectMemory() here (which will be set to the value of MaxDirectMemorySize rather than the inferred io.netty.maxDirectMemory) instead of e.g. DIRECT_MEMORY_LIMIT (which is set to the explicit or inferred value of io.netty.maxDirectMemory)?
It's possible I've misunderstood something along the way and I'm not necessarily complaining, just trying to understand: is this intended behavior? An oversight/bug? Something that is difficult to change because folks now rely on the existing semantics? Something else?