Missing data when using BeautifulSoup to find Xbox Game Pass games

user_14188553 0
Viewed 149

This page contains a list of all of the games available on Xbox Game Pass. I'd like to use BeautifulSoup to retrieve a list of the game names.

This is what I'm doing:

import requests
from bs4 import BeautifulSoup

page = requests.get('https://www.xbox.com/en-US/xbox-game-pass/games?=pcgames')

if (page.status_code != 200):
    print("Unable to load game pass games page")
    exit()

soup = BeautifulSoup(page.content, 'html.parser')
soup.find_all('h3', class_='gameDevLink') # returns []

s = soup.prettify()

with open('dump.html', 'w', encoding='utf-8') as f:
    f.write(s)

If I inspect a game on the page I see something like this:

Inspecting a game's html

Each game is enclosed in an 'a' tag with its class set to gameDevLink.

My problem is that soup.find_all('a', class_='gameDevLink') returns no hits.

If I save the html generated by BeautifulSoup to disk and search for gameDevLink there are, again, no hits.

I don't understand why I can see the information in my browser but BeautifulSoup doesn't seem to see it.

beautifulsoup python
1 Answers

The info about games is loaded from other URL via Javascript. You can use this script to simulate it:

import json
import requests
from bs4 import BeautifulSoup


headers = {'User-Agent': 'User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36'}
game_ids_url = 'https://catalog.gamepass.com/sigls/v2?id=fdd9e2a7-0fee-49f6-ad69-4354098401ff&language=en-us&market=US'
game_info_url = 'https://displaycatalog.mp.microsoft.com/v7.0/products?bigIds={}&market=US&languages=en-us&MS-CV=XXX'

game_ids = requests.get(game_ids_url).json()
s = ','.join(i['id'] for i in game_ids if 'id' in i)
    
data = requests.get(game_info_url.format(s)).json()

# uncomment this to print all data:
#print(json.dumps(data, indent=4))

# print some data to screen:
for p in data['Products']:
    print(p['LocalizedProperties'][0]['ProductTitle'])
    print(p['LocalizedProperties'][0]['ShortDescription'])
    print('-' * 80)

Prints:

A Plague Tale: Innocence - Windows 10
Follow the grim tale of young Amicia and her little brother Hugo, in a heartrending journey through the darkest hours of history. Hunted by Inquisition soldiers and surrounded by unstoppable swarms of rats, Amicia and Hugo will come to know and trust each other. As they struggle to survive against overwhelming odds, they will fight to find purpose in this brutal, unforgiving world.
--------------------------------------------------------------------------------
Age of Empires Definitive Edition
Age of Empires, the pivotal real-time strategy game that launched a 20-year legacy returns with modernized gameplay, all-new 4K visuals, 8-person multiplayer battles and a host of other new features.  Welcome back to history.
--------------------------------------------------------------------------------
Age of Empires II: Definitive Edition
Age of Empires II: Definitive Edition celebrates the 20th anniversary of one of the most popular strategy games ever with stunning 4K Ultra HD graphics, a new and fully remastered soundtrack, and brand-new content, “The Last Khans” with 3 new campaigns and 4 new civilizations.

Choose your path to greatness with this definitive remaster to one of the most beloved strategy games of all time.
--------------------------------------------------------------------------------
Age of Empires III: Definitive Edition
Age of Empires III: Definitive Edition completes the celebration of one of the most beloved real-time strategy franchises with remastered graphics and music, all previously released expansions and brand-new content to enjoy for the very first time. 
--------------------------------------------------------------------------------
Age of Wonders: Planetfall
Emerge from the cosmic dark age of a fallen galactic empire to build a new future for your people. Age of Wonders: Planetfall is the new strategy game from Triumph Studios, creators of the critically acclaimed Age of Wonders series, bringing all the exciting tactical turn-based combat and in-depth empire building of its predecessors to space in an all-new, sci-fi setting.
--------------------------------------------------------------------------------

...and so on (Total 204 games)
user_100359850
Related