OutOfDirectMemoryError using Spring-Data-Redis with Lettuce in a multi-threading context

user_2060873 0
Viewed 3510

We are using spring-data-redis with the spring-cache abstraction and lettuce as our redis-client. Additionally we use multi-threading and async execution on some methods.

An example workflow would look like this:

Main-Method A (main-thread) --> calls Method B (@Async) which is a proxy-method to be able to run the logic asynchronously in another thread. --> Method B calls Method C, which is @Cacheable. The @Cacheable Annotation handles reading/writing to our redis-cache.

What's the problem?

Lettuce is Netty-based which works relying on the DirectMemory. Due to the @Async nature of our program, we have multiple threads using the LettuceConnection (and therefore Netty) at the same time.

By design all threads will use the same (?) Netty which shares the DirectMemory. Due to an apparently too small MaxDirectMemorySize we get an OutOfDirectMemoryError, when too many threads are accessing Netty.

Example:
io.lettuce.core.RedisException: io.netty.handler.codec.EncoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 8388352 byte(s) of direct memory (used: 4746467, max: 10485760)

What have we found so far?

We use the https://docs.cloudfoundry.org/buildpacks/java/ and calculate the MaxDirectMemorySize using the https://github.com/cloudfoundry/java-buildpack-memory-calculator.

Which leeds to a MaxDirectMemorySize=10M. Having an actual available memory of 4GB the calculated MaxDirectMemorySize is probably way to conservative. This might be part of the problem.

Potential solutions to the problem

  • increase the MaxDirectMemorySize of the JVM --> but we are not sure that is sufficent
  • configure Netty not to use the DirectMemory (noPreferDirect=true) --> Netty will then use the heap, but we are insecure if this would slow down our application too much if Netty is too hungry for memory
  • no idea if this would be an option or even make the problem worse: configure Lettuce to shareNativeConnection=false --> which would lead to multiple connections to the redis

Our Question is: How do we solve this the correct way?

I'll happily provide more information on how we set up the configuration of our application (application.yml, LettuceConnection etc.), if any of these would help to fix the problem.

spring-data-redis lettuce redis asynchronous multithreading
1 Answers

Thanks to the folks over at: https://gitter.im/lettuce-io/Lobby we got some clues on how to approach these issues.

As suspected, the 10M MaxDirectMemorySize is too conservative considering the total available memory.

Recommendation was to increase this value. Since we don't actually know how much memory Netty would need to perform more stable, we thought of the following steps.

First: We will disable Netty's preference of the MaxDirectMemory by setting noPreferDirect=true. Netty will then use the heap-buffer.

Second: We will then monitor how much heap-memory Netty is going to consume during operation. Doing this, we'll be able to infer an average memory consumption for Netty.

Third: We will take the average memory consumption value and set this as the "new" MaxDirectMemorySize by setting it in the JVM option -XX:MaxDirectMemorySize. Then we'll re-enable Netty to use the DirectMemory by setting noPreferDirect=false.

Fourth: Monitor log-entries and exceptions and see if we still have a problem or if this did the trick.

[UPDATE] We started with the mentioned steps but realized, that setting noPreferDirect=true does not completely deter netty from using DirectMemory. For some use-cases (nio-Processes) Netty still uses DirectMemory.

So we had to increase the MaxDirectMemorySize.

For now we set the following JAVA_OPTS -Dio.netty.noPreferDirect=true -XX:MaxDirectMemorySize=100M. Which will probably fix our issue.

user_20608730
Related