Play framework: JsonNode streaming/chunking

Question

I have a large map that I want to return to the frontend. Originally I was converting the map to a jackson json node and returning the map back to the user with the return ok() method that play provides.

Original code:

public Result returnResponse() {
    ObjectMapper mapper = new ObjectMapper();
    Map returnMap = populateMapWithData();
    JsonNode response = mapper.valueToTree(returnMap);
    return ok(response);
}

Since the map can be really large, I am running into memory issues.

On looking at the play framework documentation, there are two ways to return large data to the frontend. If the size is known I can stream the data back to the user. If the size is not known, I can provide the data in chunks.

Play Framework documentation: https://www.playframework.com/documentation/2.8.x/JavaStream

For streaming:

public Result index() {
  java.io.File file = new java.io.File("/tmp/fileToServe.pdf");
  java.nio.file.Path path = file.toPath();
  Source source = FileIO.fromPath(path);

  Optional contentLength = null;
  try {
    contentLength = Optional.of(Files.size(path));
  } catch (IOException ioe) {
    throw new RuntimeException(ioe);
  }

  return new Result(
      new ResponseHeader(200, Collections.emptyMap()),
      new HttpEntity.Streamed(source, contentLength, Optional.of("text/plain")));
}

For chunking:

public Result index() {
  // Prepare a chunked text stream
  Source source =
      Source.actorRef(256, OverflowStrategy.dropNew())
          .mapMaterializedValue(
              sourceActor -> {
                sourceActor.tell(ByteString.fromString("kiki"), null);
                sourceActor.tell(ByteString.fromString("foo"), null);
                sourceActor.tell(ByteString.fromString("bar"), null);
                sourceActor.tell(new Status.Success(NotUsed.getInstance()), null);
                return NotUsed.getInstance();
              });
  // Serves this stream with 200 OK
  return ok().chunked(source);
}

My questions are:

How do I get the same result for a Jackson json node?
Is there another way to approach large dataset issues with the play framework?|
Do we have other play framework documentation for JSON streaming?

user_19187445 · Answer

By my opinion you must implement stream code design f.e.:

function populateMapWithData() must return stream of data, one returned element - one json node (or array of nodes, i think werry little nodes can made you response very slow)
function populateMapWithData() must get data from database by piece (when stream will try get next part of data), for save you memory
for result you must use Chunked responses because you don't know real size of content, more details in PlayFramwork Doc same link that in you question
in client side you can read every chunk or chunks array, because this approved of json specification

I lets tried make different design sample:

Results.ok().chunked( //chunked result
   Source.fromPublisher(s -> // publisher read all data by chunks
      mapper.valueToTree(populateEntryWithData(s))));

Object populateEntryWithData(Subscriber s) {
   //todo implement page reading for data
   //todo or you can try use reactive driver for database
}

ATTENTION
In play framework for working with json usually will be better use play.libs.Json or inject ObjectMapper or inject ObjectMapperBuilder.
If you need add more configuration for object mapper you can call mapper.copy(), else you can corrupt another internal mapping function in service

Play framework: JsonNode streaming/chunking

1 Answers