GET request returns 403 status code (parsing)

Question

I'm trying to get correct body of https://www.avito.ru/moskva page with status 200.

req, err := http.NewRequest("GET", "https://www.avito.ru/moskva", nil)
req.Header.Add("User-Agent", "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0")
req.Header.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")

client := &http.Client{}
res, err := client.Do(req)
if err != nil {
    panic(err)
}
defer res.Body.Close()

fmt.Println(res.Status)
printBody(res) // prints body of page

the output:

403 Forbidden
"security stub from site (says that my ip banned)"

I can open this page in browser without any warnings.

I successfully got body with python:

import requests

session = requests.Session()
session.headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0',
    'Accept-Language': 'ru',
}
print(session.get("https://www.avito.ru/moskva").text)

curl also works well, even without adding any headers:

curl https://www.avito.ru/moskva

user_2614364 · Answer

It seems the problem is the TLS version used, setting the max version to 1.2 tls.VersionTLS12 seems to work :

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "crypto/tls"
)

func main() {
    tr := &http.Transport{
        TLSClientConfig: &tls.Config{
            MaxVersion: tls.VersionTLS12,
        },
    }
    client := &http.Client{Transport: tr}
    req, err := http.NewRequest("GET", "https://www.avito.ru/moskva", nil)
    resp, err := client.Do(req)
    if err != nil {
        fmt.Println(err)
    }
    body, err := ioutil.ReadAll(resp.Body)
    bodyString := string(body)
    fmt.Print(bodyString)
}

If you switch to tls.VersionTLS13 it gives 403 status code so I'm guessing that version is negotiated by default for this host. On Chrome you can see that it's using tls1.3 :

But I'm not sure why it would return different results for tls1.3 and tls1.2

GET request returns 403 status code (parsing)

1 Answers