cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 challenge. Error when I used cloudscraper module with python

user_13136438 0
Viewed 12175

so I'm trying to bypass the cloudflare protection of a website to scrape some items from them but the Cloudscraper python module is not working.

Whenever I run it, I receive this error:

cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 challenge, This feature is not available in the opensource (free) version.

Here is a simplified code I'm using:

import cloudscraper
from bs4 import BeautifulSoup as soup


url = "http://adventurequest.life/"
scraper = cloudscraper.create_scraper()
html = scraper.get(url).text
page_soup = soup(html, "html.parser")
print(page_soup)

Do you guys have any idea how to fix this?

cloudflare beautifulsoup python
1 Answers

I encountered the same error when using scrapy + cloudscraper, but then I seted cookie_enable=true just fine：

Error

Traceback (most recent call last):
cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 Captcha challenge, This feature is not available in the opensource (free) version.
2021-04-27 09:59:30 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.forever21.com/us/shop/catalog/category/f21/lingerie>
Traceback (most recent call last):
StopIteration: <403 
https://www.forever21.com/us/shop/catalog/category/f21/lingerie>

before：

import cloudscraper

browser = cloudscraper.create_scraper()

# in middleware
req = spider.browser.get(url,
                         proxies={'http': proxy,
                                  'https': https_proxy
                                  headers={'referer': url},
                         )

after:

'COOKIES_ENABLED': True

but in bs4 Cookies are added by default， so i I tried your code and found it is normal.

url = "http://adventurequest.life/"
scraper = cloudscraper.create_scraper()
html = scraper.get(url).text
page_soup = soup(html, "html.parser")
print(page_soup)

<!DOCTYPE doctype html>
<html lang="en" style="min-height: 100%;">
<head>
<!-- Required meta tags -->
<meta charset="utf-8"/>
<meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/>
<meta content="Auto Quest Worlds" name="twitter:title"/>
<meta content="aqw bots, adventure quest bots, aqw cheat, aqw hack, aqw exploits, grimoire download, adventure quest worlds bot, leveling bot aqw, botting mmorpg, aqw private server, aqworlds private server, aqw server, aqw ps, aqw private, skidson, aqw pirata, servidor de aqw, adventure quest worlds private, dragonfable private server, adventure quest private server, free to play mmorpg, free online games, browser games, jogos online, jogos criancas, jogos de navegador, best aqw private server, best online mmorpg, best browser mmorpg, habbo servidor privado, habbo retro, habbo private server, runescape private server, high rates aqw, aqw items, aqworlds wiki" name="keywords"/>
<meta content="https://adventurequest.life/" name="twitter:url"/>

maybe you should check your machine opennssl version then updated or upgrade cloudscraper version.

my cloudscraper version is： cloudscraper ========> 1.2.58

user_146791280
Related