I am trying to scrape multiple pages using beautfiul soup but the code keeps returning the same data for every page

user_17064238 0
Viewed 82

I am trying to scrape the special offers on the steam website using Python and beautiful soup. I am trying to scrape data from mutiple pages using a for loop. I have attached the Python code below. Any help is really appreciated. Thanks in advance.

 game_lis = set([])

    for page in range(0,4):
        page_url = "https://store.steampowered.com/specials#p=" +str(page)+"&tab=TopSellers"
        #print(page_url)
        steam_games = requests.get(page_url)
        soup = BeautifulSoup(steam_games.text, 'lxml')
        s_game_offers = soup.findAll('a', class_='tab_item')
        print(page_url)

        for game in s_game_offers:
            title = game.find('div',class_='tab_item_name')
            discount = game.find('div',class_='discount_pct')
            game_lis.add(title.text)
            print(title.text+":"+discount.text)
data-analysis web-scraping beautifulsoup python
1 Answers

The page is loaded from different URL via JavaScript, so beautifulsoup doesn't see it. You can use next example how to load different pages:

import requests
from bs4 import BeautifulSoup

api_url = "https://store.steampowered.com/contenthub/querypaginated/specials/TopSellers/render/"

params = {
    "query": "",
    "start": "0",
    "count": "15",
    "cc": "SK",  # <-- probably change code here
    "l": "english",
    "v": "4",
    "tag": "",
}

for page in range(0, 4):
    params["start"] = 15 * page

    steam_games = requests.get(api_url, params=params)
    soup = BeautifulSoup(steam_games.json()["results_html"], "lxml")
    s_game_offers = soup.findAll("a", class_="tab_item")

    for game in s_game_offers:
        title = game.find("div", class_="tab_item_name")
        discount = game.find("div", class_="discount_pct")
        print(title.text + ":" + discount.text)

    print("-" * 80)

Prints:

F.I.S.T.: Forged In Shadow Torch:-10%
HITMAN 2 - Gold Edition:-85%
NieR:Automata™:-50%
Horizon Zero Dawn™ Complete Edition:-40%
Need for Speed™ Heat:-86%
Middle-earth: Shadow of War Definitive Edition:-80%
Batman: Arkham Collection:-80%
No Man's Sky:-50%
Legion TD 2 - Multiplayer Tower Defense:-20%
NieR Replicant™ ver.1.22474487139...:-35%
Days Gone:-20%
Mortal Kombat 11 Ultimate:-65%
Human: Fall Flat:-66%
Muse Dash - Just as planned:-30%
The Elder Scrolls Online - Blackwood:-50%
--------------------------------------------------------------------------------
The Elder Scrolls Online - Blackwood:-50%
Football Manager 2022:-10%
Age of Empires II: Definitive Edition:-33%
OCTOPATH TRAVELER™:-50%
DRAGON QUEST® XI S: Echoes of an Elusive Age™ - Definitive Edition:-35%
Witch It:-70%
Monster Hunter: World:-34%
NARUTO SHIPPUDEN: Ultimate Ninja STORM 4:-77%
MADNESS: Project Nexus:-10%
Mad Max:-75%
Outer Wilds:-40%
Middle-earth: Shadow of Mordor Game of the Year Edition:-75%
Age of Empires III: Definitive Edition:-33%
Ghostrunner:-60%
The Elder Scrolls® Online:-60%
--------------------------------------------------------------------------------

...
user_100359850
Related