How to do multiple paginated requests concurrently at once and collect in order with Spring WebFlux

user_1589188 0
Viewed 62

I am trying to send multiple paginated requests and return a Flux<Item>, the requests can be done in parallel but the results have to be collected in order.

public Mono<PageOfItem> getAPageOfItems(int page) {
    // API call to return a page of items from the given page
}

public class PageOfItem {
    List<Item> items;  // the list of items
    int page;          // the current page number
    int totalPage;     // the total number of pages
}

What I have done is like this:

public Flux<Item> chainPageOfItems(int page, List<Item> allItems) {
    return getAPageOfItems(page).flatMap(pageOfItem -> {
        allItems.addAll(pageOfItem.items);
        if (page < pageOfItem.totalPage) {
            return chainPageOfItems(page + 1, allItems);
        }
        return Flux.fromIterable(allItems);
    });
}

Obviously the above doesn't work, just my failed attempt, so seeking help here on SO. Thank you!

project-reactor spring-webflux java
1 Answers

flatMapSequential will retrieve pages concurrently while keeping the order:

public Flux<Item> chainPageOfItems() {
    return getAPageOfItems(1)
        .flatMapMany(firstPage ->
            Flux.concat(
                Flux.fromIterable(firstPage.getItems()),
                Flux.range(2, firstPage.getTotalPage() - 1)
                    .flatMapSequential(this::getAPageOfItems)
                    .flatMapIterable(PageOfItem::getItems)
            ));
}
user_60511760
Related