How to bypass cloudflare using Scrapy?

user_5973623 0
Viewed 3198

Is there any simple way to bypass cloudflare protection with Scrapy? I tried cloudflare-scrape but it doesn't work for me and still get 403 error. Thanks.

scrapy-spider cloudflare
2 Answers

The 403 error is usually related to your userAgent. Try setting your userAgent to some common browser's userAgent (e.g. 'Mozilla').

user_15587310

If you're getting 403 Error you can follow these guidlines and if you're facing something else let me know the exact error, i will help.

Go to settings.py

Search: USER_AGENT

Here you will see the default bot user agent by scrapy.

Replace that default with this:

USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'

You will see 503 is no more.

user_125932270
Related