BeautifulSoup returning None during site scraping

user_17589544 0
Viewed 56

I'm trying to scrape Origin site to check if given game title is available for MacOS. For example The Sims 4 (https://www.origin.com/pol/pl-pl/store/the-sims/the-sims-4). There's Apple logo icon, which I believe would be great indicator to check for. So I've looked into code and found it:
<i ng-repeat="platform in platforms track by $index" class="otkicon otkicon-apple"></i>

Now into Python code:

page = requests.get(origin_site)
soup = BeautifulSoup(page.content, "html.parser")
result = soup.find("i", class_="otkicon otkicon-apple")
if result:
    origin_answer = "YES"

Aaaaand it returns None.
For other websites (Steam, Gog, Epic) it works. I've been googling it for past 2h and tried every solution I could find. None (nomen omen) of them worked. I've also tried with headers, also nothing. I seriously got no idea what to do next.

web-scraping beautifulsoup html python
1 Answers

This website needs JS to work, and requests does not supports it. But you can:

  • If any, use an API.
  • Use requests-html which fully supports Javascript.
  • Use Selenium Python binding, which will allow you to automates your browser.
Admin1
Related