Using Rust libraries reqwest and select in conjunction

Question

I am trying to follow the example here: https://rust-lang-nursery.github.io/rust-cookbook/web/scraping.html, which utilizes both Reqwest and Select in order to get an html response then parse the data.

I am using Reqwest version 0.10.4 and Select version 0.4.3 which are the versions it shows in the example. However, I am getting an error:

error[E0277]: the trait bound `reqwest::Response: std::io::Read` is not satisfied
  --> src/main.rs:19:25
   |
19 |     Document::from_read(res)?
   |                         ^^^ the trait `std::io::Read` is not implemented for `reqwest::Response`
   | 
  ::: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/select-0.4.3/src/document.rs:31:25
   |
31 |     pub fn from_read(mut readable: R) -> io::Result {
   |                         -------- required by this bound in `select::document::Document::from_read`

It seems like the from_read method takes in a Read type but the reqwest::get method returns a different type. Is there some sort of conversion that must be done first before the response is passed to the from_read method?

This is the example:

#[macro_use]
extern crate error_chain;
extern crate reqwest;
extern crate select;

use select::document::Document;
use select::predicate::Name;

error_chain! {
   foreign_links {
       ReqError(reqwest::Error);
       IoError(std::io::Error);
   }
}

fn main() -> Result<()> {
    let res = reqwest::get("https://www.rust-lang.org/en-US/").await?;

    Document::from_read(res)?
        .find(Name("a"))
        .filter_map(|n| n.attr("href"))
        .for_each(|x| println!("{}", x));

    Ok(())
}

user_9363973 · Answer

reqwest::get returns a Result, then with the ? you are unwrapping the Result, meaning you now have a Response object as documented here. And because a web call can successfully occur but still fail (see HTTP non 200 codes) you should check the response code, but since this is to learn we'll ignore it. What you want is a struct that implements the std::io::Read trait, reading up on that shows that String implements that trait. Going back to reqwest::Response shows that we can get the string returned using the method text(). So your code now becomes

let res = reqwest::get("https://www.rust-lang.org/en-US/")
              .await?
              .text()
              .await?;

Update from the comments: Now the problem is that Document::from_read only accepts std::io::Read as parameters, and std::string::String does not implement that trait, and instead of using a crate like stringreader we can simply use Document::from as document implements the From<&'a str> trait.

And, as with almost everything, there are multiple ways of solving this. You could either

Create the Document from the string directly with Document::from(res), or
Convert the string to a byte slice, which implements Read, and create the document from that with Document::from_read(res.as_bytes())

Using Rust libraries reqwest and select in conjunction

1 Answers