Why am I getting Error 429 from a specific website on fetch() using Scrapy?

user_7162306 0
Viewed 903

I'm very new to web scraping and and as a first project(in order to learn) I wanted to create a database for house prices. Later on I'm going to feed it to ML algorithms to see if I'm going to be able to predict the prices but I cannot fetch the page. I'm getting this:

In [1]: fetch("https://www.sahibinden.com")
2020-11-07 01:37:34 [scrapy.core.engine] INFO: Spider opened
2020-11-07 01:37:34 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.sahibinden.com> (failed 1 times): 429 Unknown Status
2020-11-07 01:37:34 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.sahibinden.com> (failed 2 times): 429 Unknown Status
2020-11-07 01:37:34 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.sahibinden.com> (failed 3 times): 429 Unknown Status
2020-11-07 01:37:34 [scrapy.core.engine] DEBUG: Crawled (429) <GET https://www.sahibinden.com> (referer: None)

The last Crawled (429) message yields an error page. That's obviously not the page I'm looking for. I'm getting 200 from any other website. Only this website is problematic. Is there a way to fix this?

scrapy-shell scrapy web-scraping python
3 Answers

429 HTTP status code means too many requests. Your requests to this site has been reached to limits. Many services define request per second limit to avoid DOS. You have to pause between your requests. But how long? You need to try more to estimate appropriate sleeping/pausing time. Pausing could be defined after each request or after a bunch of requests. You can use time.sleep() for pausing.

user_131183270
user_111089660

If you are new to web scraping you have to know that HTTP 429 is going to be your new friend.

However, I've found out a nice workaround to IP blocking when scraping sites. It lets you run a Scraper indefinitely by running it from Google App Engine and redeploying it automatically when you get a 429.

Check out my article here

user_59826370
Related