Reduce List<CompletableFuture<T>>

user_1136878 0
Viewed 1102

When ints is given:

List<Integer> ints = IntStream.range(0, 1000).boxed().collect(Collectors.toList());

With Java Stream API, we can reduce them

MyValue myvalue = ints
        .parallelStream()
        .map(x -> toMyValue(x))
        .reduce((t, t2) -> t.combine(t2))
        .get();

In this example, what important to me are...

  • items will be reduced in multiple threads
  • early mapped items will be reduced early
  • not all result of toMyValue() will be loaded at the same time

Now I want to do same processing by CompletableFuture API.

To do map, I did:

List<CompeletableFuture<MyValue>> myValueFutures = ints
        .stream()
        .map(x -> CompletableFuture.supplyAsync(() -> toMyValue(x), MY_THREAD_POOL))
        .collect(Collectors.toList());

And now I have no idea how to reduce List<CompeletableFuture<MyValue>> myValueFutures to get single MyValue.

Parallel stream provides convenient APIs but because of these problems I want not to use Stream API:

  • parallel stream is hard to stop the stage during processing.
  • parallel stream's active worker count can exceed the parallelism when some workers blocked by IO. This helps to maximize cpu utilization but memory overhead can occur(even OOM).

Any way to reduce CompetableFutures? one by one with out stream reduce api?

completable-future java-stream java
3 Answers

It's interesting that in your initial example you already mention a method you have combine, while for CompletableFuture there is a dedicated method, just for that : thenCombine (and its two brothers thenCombineAsync).

So considering that you have something like:

static class MyValue {
    final int x;

    MyValue(int x) {
        this.x = x;
    }

    MyValue combine(MyValue v){
        return new MyValue(this.x + v .x);
    }
}

static MyValue toMyValue(int x) {
    return new MyValue(x);
}

And:

List<CompletableFuture<Integer>> list = 
    IntStream.range(0, 4)
             .mapToObj(x -> supplyAsync(() -> x))
             .collect(Collectors.toList());

You could use one of the thenCombine methods and achieve what you want via:

MyValue value =
    list.stream()
        .map(x -> x.thenApply(YourClass::toMyValue))
        .reduce((left, right) -> left.thenCombine(right, MyValue::combine))
        .orElse(CompletableFuture.completedFuture(new MyValue(0)))
        .join();

If you want to execute the combine action in a predictable thread, you need a pool for that, or an overloaded method, like:

.reduce((left, right) -> left.thenCombineAsync(right, MyValue::combine, MY_THREAD_POOL))
user_10593720

Basically you need to wait for the results of all the CompletableFutures and combine then to obtain the result required.

There are several approaches for that, but the CompletableFuture class provides the method allOf that can be used for that purpose.

When I have to do with a similar problem I like to follow the Tomasz Nurkiewicz advice and perform this kind of computation in the following way.

As suggested in the article, first, let's define the following convenient method: allOf receives arguments as varargs and doesn't return a future of aggregated results; this method will allow you to overcome those drawbacks so you can pass a Collection as argument and return the List of actual results instead of Void.

private static <T> CompletableFuture<List<T>> sequence(List<CompletableFuture<T>> futures) {
    CompletableFuture<Void> allDoneFuture =
        CompletableFuture.allOf(futures.toArray(new CompletableFuture[futures.size()]));
    return allDoneFuture.thenApply(v ->
            futures.stream().
                    map(future -> future.join()).
                    collect(Collectors.<T>toList())
    );
}

With this handy method in place you can reduce your values with something like:

final CompletableFuture<List<MyValue>> allDone = sequence(myValueFutures);

// Please, see also for alternate approaches
// https://stackoverflow.com/questions/43489281/return-value-directly-from-completablefuture-thenaccept
final List<MyValue> myValues = allDone.join();

final Optional<MyValue> optResult = myValues.stream().
  reduce((t, t2) -> t.combine(t2))
;

// Process the returned value as you consider appropriate
final MyValue result = optResult.get();
user_139424480

This is not a direct answer to your question of going from List<CompeletableFuture<MyValue>> to a single MyValue

There's a library com.spotify:completable-futures which can easily reduce a List<CompletableFuture<T>> to CompletableFuture<List<T>>. That might be the most common use case.

https://github.com/spotify/completable-futures

user_1660620
Related