Scraping Stream via Beautiful Soup, banner art is empty 'trans.gif' file

user_16913355 0
Viewed 73

I'm currently writing some Python code that scrapes the Steam homepage with Beautiful Soup, and outputs some basic information about the games listed.

page_soup = soup(page_html, 'html.parser')
container = page_soup.findAll('a', {'class':'tab_item'})
container.append(page_soup.findAll('a', {'class':'tab_item.app_impression_tracked'}))

[...]

    count = 0
    for item in container:
        price = container[count].find('div','tab_item_discount')
        title = container[count].find('div','tab_item_content')
        cover = container[count].find('div', 'tab_item_cap')
        tags = title.find('div', 'tab_item_top_tags')
        print("price: " + price['data-price-final'])
        print("Title: " + title.div.text)
        print("Cover: " + cover.img['src'])
        print("Tags: " + tags.text)
        count += 1

Which outputs:

price: 0
Title: RetroArch
Cover: https://store.akamai.steamstatic.com/public/shared/images/trans.gif
Tags: Free to Play, Retro, Singleplayer, Multiplayer
price: 5999
Title: DEATHLOOP
Cover: https://store.akamai.steamstatic.com/public/shared/images/trans.gif
Tags: Action, FPS, First-Person, Stealth
[...]

This mostly works, except the cover (banner image) is being grabbed as an empty 1x1 'trans.gif' file. I'm unsure what the flaw in my code here is that's causing this.

steam beautifulsoup python
1 Answers

It's not a flaw in your code. That 1x1 image is probably a placeholder that gets replaced later by Javascript on the page. BeautifulSoup doesn't execute Javascript. If you really need that, you'll need to use something like Selenium to run a Chrome instance. It's more trouble, so make sure you need that cover art.

user_18833160
Related