I receive request body from client as sampled below -
headerrowcolumn 1 | headerrowcolum 2 | headerrowcolumn 3|.....|headerrowcolumn n|
row1value1 | row1value2 | row1value3 | .....|row1valuen|
.
.
.
rowNvalue1 | rowNvalue2 | rowNvalue3 | .....|rowNvaluen|
The data contains more than 100k rows and I want to be able to save them one by one.
Currently I am reading all the data and converting it to a Flux of corresponding POJO. But this means I need to read all the data and store it until I am done persisting it to the DB. This takes a lot of time and hogs around 5gb of MEM
How can I read this one line at a time and then save it one Mono at a time? Thank you!
My current code (Handler class) is as follows:
public Mono<ServerResponse> saveLargeDataset(final ServerRequest request){
return request.bodyToMono(String.class)
.flatMap(body -> {
//pass in the request body and map each line to the POJO
info = serviceClass.getFluxOfData(body);
//send the flux of POJO to dataManager
dataManagerClass.saveData(info)
.doOnError(error -> "do something on error").subscribe();
return ServerResponse.ok().build();
});
}