Webclient Leaks Memory In Docker?

Viewed 2366

I'm trying to use Webclient in my project, but when I load test, I'm noticing the docker memory usage never goes down until the instance dies.

@Component
public class Controller {

  //This is an endpoint to another simple api
  //I use my local Ip instead of localhost in the container
  private static final String ENDPOINT = "http://localhost:9090/";

  private WebClient client;
  
  public Controller(WebClient.Builder client) {
    super();
    this.client = client.build();
  }

  @Bean
  public RouterFunction<ServerResponse> router() {

    return RouterFunctions.route(GET("helloworld"), this::handle);
  }

  Mono<ServerResponse> handle(ServerRequest request) {

    Mono<String> helloMono =
        client.get().uri(ENDPOINT + "/hello").retrieve().bodyToMono(String.class);

    Mono<String> worldMono =
        client.get().uri(ENDPOINT + "/world").retrieve().bodyToMono(String.class);

    return Mono.zip(helloMono, worldMono, (h, w) -> h + w)
        .flatMap(s -> ServerResponse.ok().bodyValue(s));
  }
}

Here's my dockerFile as well.

FROM openjdk:8

ENV SERVICE_NAME reactive-hello-world

ADD target/reactive-hello-world-*.jar $APP_HOME/reactive-hello-world.jar

RUN mkdir /opt/reactor-netty/


EXPOSE 9010

CMD java \
    -Dcom.sun.management.jmxremote=true \
    -Dcom.sun.management.jmxremote.local.only=false \
    -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.ssl=false \
    -Djava.rmi.server.hostname=localhost \
    -Dcom.sun.management.jmxremote.port=9010 \
    -Dcom.sun.management.jmxremote.rmi.port=9010 \ 
    -Xmx190M \
    -jar reactive-hello-world.jar

EXPOSE 8080

Have I missed a step somewhere?

Edit: Here's some images

Before Load Test: Before load test

After Load Test

enter image description here

As you can see, the GC is happening correctly but the memory hasn't decreased. If I let the test continue it kills the instance in a couple minutes.

I've tried similar code using RestTemplate and I'm not experiencing any issues, the memory doesn’t usually exceed 400MB even when I run the Jmeter for an extended time. Can you help understand what's happening?

Edit: I've also tried the deprecated AsyncRestTemplate and I'm not seeing a problem with that either.

Edit: I have created the repos for this example. Please check if you can reproduce the issue.

The Hello World Backend

The Webclient Hello World(JMX is inside this repo)

The RestTemplate Hello World

The AsyncRestTemplate Hello World

2 Answers

I don't think your problem has anything to do with the RestTemplate vs WebClient. Your GC graph looks pretty normal. It doesn't look like there is a leak, because whenever a GC happens, it seems like the allocated memory is able to go back to the previous levels.

It is important to note that you wouldn't always see a drop in the container memory usage when a garbage collection happens. This is because the Java Virtual Machine does not necessarily return memory back to system after a GC. This means, even if a portion of the memory is unused/freed after a GC in your process, it may still appear as "used" from outside of the process.

To be clear: JVM does return memory back to the system in some circumstances, and it depends on various factors including the garbage collector that is used.

From the GC graph on the second screenshot of your example, the GC graph looks pretty normal, but it seems that a pretty small amount of heap was released back to the system (the orange area).

You can try switching to G1 GC via -XX:+UseG1GC JVM flag and tune the behavior to more aggressively release the unallocated memory back to the system by tweaking -XX:MinHeapFreeRatio and -XX:MaxHeapFreeRatio, particularly by reducing the -XX:MaxHeapFreeRatio value from the default 70 percent to a lower number. See here.

Lowering -XX:MaxHeapFreeRatio to as low as 10% and -XX:MinHeapFreeRatio has shown to successfully reduce the heap size without too much performance degradation; however, results may vary greatly depending on your application. Try different values for these parameters until they're as low as possible, yet still retain acceptable performance.

For further information on the topic, see:

Does GC release back memory to OS?

JEP 346: Promptly Return Unused Committed Memory from G1

Related