Spring WebClient: Parse + Stream very large JSON

user_8198203 0
Viewed 1713

This question is similar to Spring reactive streaming data from regular WebClient request with the difference that I'm not getting JSON array immediately from my WebClient, but something like this:

This JSON object can be very large (~100MB), and thus needs to be worked on and streamed to the client, instead of parsed. This here is the only way I seem to be able to get the semantics correct:

{
   "result-set":{
      "docs":[
         {
            "id":"auhcsasb1005_100000"
         },
         {
            "id":"auhcsasb1005_1000000"
         },
         {
            "id":"auhcsasb1005_1000001"
         },
         {
            "id":"auhcsasb1005_1000002"
         },
         ...
         ...
         {
            "EOF":true
         }
      ]
   }
}

WebClient.create()
  .get()
  .retrieve()
  .bodyToMono(DontKnowWhatClass.class)
  .flatMapMany(resultSet -> Flux.fromIterable(resultSet.getDocs()))

BUT that means that I'm deserializing 100MB or more in memory, to then create a flux from it. What I'm wondering is: Am I missing something crucial? Can I somehow just create a Flux from an Object like that? I have now way to influence how the result-set object is rendered, sadly.

spring-webflux
2 Answers

I cannot imagine a way to reliably parse such a huge chunk of json in smaller parts. You could try to somehow convert the big chunk into smaller tokens and try to process them step by step.

But I would assume that this ending with the expected result, i.e. allow a more memory efficient parsing, is absolutely not guaranteed.

But, there are other ways to approach this problem, especially when you work reactive.

If you work in a reactive WebFlux context, you could try to use backpressure or rate limiting to make your application only parse a limited number of JSON objects at the same time. This would preserve the limited resources (RAM, CPU, JVM threads etc.) of your application.

And if the size of the objects really passes the 100MB limit, you should really consider questioning the current data model. Is this data structure and its size really suitable?

Maybe this problem cannot be solved with technical / implementation means. Maybe the current application design need to be changed.

user_29911050

You can accept a ServerWebExchange to your controller which has a method that will take a Publisher exchange.response.writeWith().

If you have a way to parse the payload in chunks you just create a Flux that emits the parts.

For example, if you don't care about the payload at all and just want to ship it as-is:

    @GetMapping("/api/foo/{myId}")
    fun foo(exchange: ServerWebExchange, @PathVariable myId: Long): Mono<Void> {
        val content: Flux<DataBuffer> = webClient
            .get()
            .uri("/api/up-stream/bar/$myId")
            .exchange()
            .flatMapMany { it.bodyToFlux<DataBuffer>() }

        return exchange.response.writeWith(content)
    }

Make sure you check the content negotiation settings to avoid something buffering you didn't expect.

user_53145460
Related