python web scraper, Response [403]

Viewed 313

I'm trying to web scrape a website but I got the following error :

<Response [403]>

When I display the output I got this :

<body style="margin:0"><p id="cmsg">Please enable JS and disable any ad blocker</p><script>var dd={\'cid\':\'AHrlqAAAAAMAFuvaKabaZaMAVsjd8A==\',\'hsh\':\'05B30BD9055986BD2EE8F5A199D973\',\'t\':\'fe\',\'s\':2089,\'host\':\'geo.captcha-delivery.com\'}</script><script src="https://ct.captcha-delivery.com/c.js"></script>

After checking on the net, I've find that we need to add a header otherwise the websit will know that is a bot. I've added it but still the error.

The code I'm using :

from IPython.display import Image
import requests
from bs4 import BeautifulSoup
import requests_cache
import pandas as pd
requests_cache.install_cache("bases_scraping", expire_after=10e5)

url = "https://www.leboncoin.fr/recherche?category=55&text=electric%20bike&locations=r_12"

headers = {"user-agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0"}

response = requests.get(url, headers=headers)
print(response)
print(response.content, "html.parser")

If someone can help me.

Thanks

1 Answers

That doesn't seem to be the problem of bot blocker, make sure that the website you are scraping doesn't use javascript to load its content and if does then you'll have to use a javascript enabled scraping library such as selenium to let the website load its content and maybe then use beautifulsoup to parse the dom.

Related