spring webflux functional how to efficiently read large data from request

Viewed 143

I receive request body from client as sampled below -

headerrowcolumn 1 | headerrowcolum 2 | headerrowcolumn 3|.....|headerrowcolumn n|
row1value1 | row1value2 | row1value3 | .....|row1valuen|
.
.
.
rowNvalue1 | rowNvalue2 | rowNvalue3 | .....|rowNvaluen|

The data contains more than 100k rows and I want to be able to save them one by one.

Currently I am reading all the data and converting it to a Flux of corresponding POJO. But this means I need to read all the data and store it until I am done persisting it to the DB. This takes a lot of time and hogs around 5gb of MEM

How can I read this one line at a time and then save it one Mono at a time? Thank you!

My current code (Handler class) is as follows:


 public Mono<ServerResponse> saveLargeDataset(final ServerRequest request){

            return request.bodyToMono(String.class)
                    .flatMap(body -> {
                        //pass in the request body and map each line to the POJO
                        info = serviceClass.getFluxOfData(body);
                        //send the flux of POJO to dataManager
                        dataManagerClass.saveData(info)
                                .doOnError(error -> "do something on error").subscribe();
                        return ServerResponse.ok().build();
                    });
        }

0 Answers
Related