Any idea how to get to this url with scrapy?

Viewed 606

The URL is "https://sb-content.pa.caesarsonline.com/content-service/api/v1/q/time-band-event-list?".

I think it uses Cloudflare which is why I am having difficulty but I am not sure if that is the only issue. I don't necessarily need a solution shown in Scrapy, I have played around with cfscrape and can't get it to get any response other than 403 as well.

1 Answers

You are correct in assuming that this is cloudflare blocking automated requests.

<title>Access denied | sb-content.pa.caesarsonline.com used Cloudflare to restrict access</title>

You can use the library "cloudscraper" to try and bypass this but as cloudflare changes their detection methods periodically you might eventually have troubles until the library is updated.

Cloud Scraper Library: https://pypi.org/project/cloudscraper/

Example:

import cloudscraper

scraper = cloudscraper.create_scraper()
response = scraper.get("https://sb-content.pa.caesarsonline.com/content-service/api/v1/q/time-band-event-list?").text
print(response)

Output:

{"data":{"timeBandEvents":[{"type":"LIVE","date":null,"competitionSummary":[],"events":[],"outrights":[]},{"type":"NEXT_TO_GO"........
Related