Why does the reqwest HTTP library return binary data instead of a text body?

user_1013173 0
Viewed 834

I am trying to perform a HTTP GET request with reqwest and print the response body to STDOUT. This works for most websites, but it returns weird binary output for amazon.com:

#[tokio::main]
async fn main() {
    run().await;
}

async fn run() {
    let url = "https://www.amazon.com/PNY-GeForce-Gaming-Overclocked-Graphics/dp/B07GJ7TV8L/";
    let resp = reqwest::get(url).await.unwrap();
    let text = resp.text().await.unwrap();
    println!("{}", text);
}

Why would resp.text().await.unwrap() return binary data and how can I obtain normal HTTP body from it?

curl returns the HTML I expected:

curl https://www.amazon.com/PNY-GeForce-Gaming-Overclocked-Graphics/dp/B07GJ7TV8L/
reqwest rust http
1 Answers

If you do curl https://www.amazon.com/PNY-GeForce-Gaming-Overclocked-Graphics/dp/B07GJ7TV8L/ - I you will see:

server: Server
content-type: text/html
content-length: 2148
content-encoding: gzip
x-amz-rid: 2T9PBCY66S79SMC424V2
vary: Accept-Encoding
akamai-cache-status: Miss
date: Sat, 29 Feb 2020 22:23:54 GMT

content-encoding: gzip it's quite obvious what you need to do. Checkout gzip from reqwest. gzip is a optional features, see cargo doc, for reqwest you can write reqwest = { version = "0.10.3", features = ["gzip"] } in your Cargo.toml.

user_70761530
Related