i want to scrape this website with python
https://animepahe.org/api?m=release&id=167&l=30&sort=episode_asc&page=1 but this site is protected by cloudflare
Checking your browser before accessing website.com.
This process is automatic. Your browser will redirect to your requested content shortly.
Please allow up to 5 seconds...
does anyone know how to bypass it
You can do it like this:
import cloudscraper
scraper = cloudscraper.create_scraper()
url = 'https://animepahe.org/api?m=release&id=167&l=30&sort=episode_asc&page=1'
r = scraper.get(url)
print(r.text)
This will handle Cloudflare protection and return the html for that page. You can install cloudscraper with pip install cloudscraper.
Animepahe is protected with Cloudflare. Cloudflare is a Web Application Firewall (WAF), which is an anti-bot service.
It uses a variety of techniques to identify you as a bot. Your client faces a number of challenges when connecting to the website, such as detecting botnets, IP reputation, HTTP request headers, TLS fingerprinting, HTTP/2 fingerprinting, event tracking and environment API tracking.
Cloudflare collects the data from all of the mentioned methods as sensor data and validates it for inconsistencies on the server side. Know more about Cloudflare's bot detection from here: How does Cloudflare detect bots?.
You can bypass the WAF or getting ignored by it using clouscraper or web browser based libraries such as undetected-chromedriver and selenium-stealth, which implements many techniques to avoid triggering the WAFs.
Unfortunately, cloudscraper doesn't work for this case anymore. It detects more recent challenges and can't solve it with the opensource version.
CloudflareChallengeError: Detected a Cloudflare version 2 challenge, This feature is not available in the opensource (free) version
You can use undetected-chromedriver in Selenium:
import undetected_chromedriver as uc
from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
import time
options = webdriver.ChromeOptions()
chrome_path = ChromeDriverManager().install()
chrome_service = Service(chrome_path)
driver = uc.Chrome(options=options, service=chrome_service, use_subprocess=True)
url = "https://animepahe.org/api?m=release&id=167&l=30&sort=episode_asc&page=1"
driver.get(url)
time.sleep(5)
#do something
driver.quit()