How to convert future::join_all() to using FuturesUnordered?

user_13915110 0
Viewed 1036

I am making something like below! function1() and function2() both are doing async steps to insert a large amount of data into the database. However, I think join_all() can't handle too much data. It always stop after I inserted about 1500 data. I saw someone say FuturesUnordered is good to handle large operations, but I didn't find any example for it. Is anyone has experience with it? If so, can show to how to convert my code to use FuturesUnordered?

async fn main() {
    println!("Start to insert data to database!");
    block_on(hold());
}

async fn hold() {
    let futures = vec![function1(),function2()];
    let _results = future::join_all(futures).await;
}

Below, is the code I did for FuturesUnordered, but not work!

async fn main() {
    println!("Start to insert data to database!");
    block_on(hold());
}

async fn hold() {
    let futures = FuturesUnordered::collect（function1(),function2()）;
    let _results = FuturesUnordered::poll(futures).await;
}
rust-tokio rust
1 Answers

You can either create it and push tasks to it manually:

let futures = FuturesUnordered::new();
futures.push(function1());
futures.push(function2());

Or if your tasks are already in an iterable, you can collect into it just like any other collection:

let futures: FuturesUnordered<_> = vec![function1(), function2()]
    .into_iter()
    .collect();

Then you can simply collect and await the results (.collect() provided by StreamExt):

let results: Vec<_> = futures.collect().await;

See these working on the playground.

I am skeptical that this change will address your root problem though. No use of join_all should cause your tasks to run incorrectly, and swapping it for FuturesUnordered should only yield a performance difference, and likely not much of one for only two tasks. The docs for join_all already say it'll switch to using FuturesUnordered internally when there are a large number of tasks, and the current threshold is set at 30.

user_21891300
Related