Unable to join threads from JoinHandles stored in a Vector - Rust

user_14770761 0
Viewed 2348

I am writing a program which scrapes data from a list of websites and stores it into a struct called Listing which is then collected into a final struct called Listings.

use std::{ thread,
           sync::{ Arc, Mutex }
         };

fn main() {
    // ... some declarations
    let sites_count = site_list.len(); // site_list is a vector containing the list of websites

    // The variable to be updated by the thread instances ( `Listing` is a struct holding the information ) 
    let listings: Arc<Mutex<Vec<Vec<types::Listing<String>>>>> = Arc::new(Mutex::new(Vec::new()));

    // A vector containing all the JoinHandles for the spawned threads
    let mut fetch_handle: Vec<thread::JoinHandle<()>> = Vec::new();

    // Spawn a thread for each concurrent website
    for i in 0..sites_count { 
        let slist = Arc::clone(&site_list);
        let listng = Arc::clone(&listings);
        fetch_handle.push(
            thread::spawn(move || {
                println!("⌛ Spawned Thread: {}",i);
                let site_profile = read_profile(&slist[i]);
                let results = function1(function(2)) // A long list of functions from a submodule that make the http request and parse the data into `Listing`
                listng.lock().unwrap().push(results);
            }));
    }
    
    for thread in fetch_handle.iter_mut() { 
        thread.join().unwrap();
    }

    // This is the one line version of the above for loop - yields the same error.
    // fetch_handle.iter().map(|thread| thread.join().unwrap()); 

    // The final println to just test feed the target struct `Listings` with the values
    println!("{}",types::Listings{ date_time: format!("{}", chrono::offset::Local::now()),
                                   category: category.to_string(),
                                   query: (&search_query).to_string(),
                                   listings: listings.lock().unwrap() // It prevents me from owning this variable
                                 }.to_json());
}

To which I stumble upon the error

error[E0507]: cannot move out of `*thread` which is behind a mutable reference
   --> src/main.rs:112:9
    |
112 |         thread.join().unwrap();
    |         ^^^^^^ move occurs because `*thread` has type `JoinHandle<()>`, which does not implement the `Copy` trait

It prevents me from owning the variable after the thread.join() for loop.

When I tried assigning to check the output type

let all_listings = listings.lock().unwrap()

all_listings reports a type of MutexGuard(which is also true inside the thread for loop, but it allows me to call vector methods on it) and wouldn't allow me to own the data. I changed the data type in the Listings struct to hold a reference instead of owning it. But it seems so the operations I perform on the struct in .to_json() require me to own its value. The type declaration for listings inside the Listings Struct is Vec<Vec<Listing<T>>.

This code however works just fine when I move the .join().unwrap() to the end of thread::spawn() block or apply to its handle inside the for loop(whilst disabling the external .join() ). But that makes all the threads execute in a chain which is not desirable, since the main intention of using threads was to execute same functions with different data values simultaneously.

I am quite new to Rust in general(been 3 weeks since I am using it) and its my first time ever implementing Multithreading. I have only ever written single threaded programs in java and python before this, so if possible be a little noob friendly. However any help is appreciated :) .

rust multithreading
2 Answers

I figured out what needed to happen. First, for this kind of thing, I agree that into_iter does what you want, but it IMO it obscures why. The why is that when you borrow on it, it doesn't own the value, which is necessary for the join() method on the JoinHandle<()> struct. You'll note its signature takes self and not &mut self or anything like that. So it needs the real object there.

To do that, you need to get your object out of the Vec<thread::JoinHandle<()>> that it's inside. As stated, into_iter does this, because it "destroys" the existing Vec and takes it over, so it fully owns the contents, and the iteration returns the "actual" objects to be joined without a copy. But you can also own the contents one at a time with remove as demonstrated below:

while fetch_handle.len() > 0 {
    let cur_thread = fetch_handle.remove(0); // moves it into cur_thread
    cur_thread.join().unwrap();
}

This is instead of your for loop above. The complete example in the playground is linked if you want to try that.

I hope this is clearer on how to work with things that can't be copied, but methods need to fully own them, and the issues in getting them out of collections. Imagine if you needed to end just one of those threads, and you knew which one to end, but didn't want to end them all? Vec<_>::remove would work, but into_iter would not.

Thank you for asking a question which made me think, and prompted me to go look up the answer (and try it) myself. I'm still learning Rust as well, so this helped a lot.

Edit:

Another way to do it with pop() and while let:

while let Some(cur_thread) = fetch_handle.pop() {
    cur_thread.join().unwrap();
}

This goes through it from the end (pop pulls it off of the end, not the front), but doesn't reallocate or move the vector contents via pulling it off the front either.

user_838390

Okay so the problem as pointed out by @PiRocks seems to be in the for loop that joins the threads.

 for thread in fetch_handle.iter_mut() {
        thread.join().unwrap();
    }

The problem is the iter_mut(). Using into_iter() instead

 for thread in fetch_handle.into_iter() {
        thread.join().unwrap();
    }

yields no errors and the program runs across the threads simultaneously as required.

The explanation to this, as given by @Kevin Anderson is:

Using into_iter() causes JoinHandle<()> to move into the for loop.

Also looking into the docs(std::iter) I found that iter() and iter_mut() iterate over a reference of self whereas into_iter() iterates over self directly(owning it).

So iter_mut() was iterating over &mut thread::JoinHandle<()> instead of thread::JoinHandle<()>.

user_147707610
Related