Scraper returns null result

user_8906587 0
Viewed 476

Im trying to build a scraper that get the URL for free games of the epic games store

enter image description here

headers = {"User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0",}
result = requests.get("https://www.epicgames.com/store/en-US/free-games?sessionInvalidated=true", 
headers=headers)
soup = BeautifulSoup(result.content, 'lxml')
urls = []
links = []
urls = soup.find('div', {'class': 'CardGrid-group_c5363b6a'}).find_all("a")
return urls

enter image description here however it keeps returning null and I cant see what is wrong?

screen-scraping python
1 Answers

This page uses JavaScript to add elements but requests/BeautifuSoup can't run JavaScript

But as usually JavaScript reads data from URL which you can find in DevTools in Firefox/Chrome (tab: Network, filter: XHR) and you can use it to read data in JSON format - so you don't need BeautifulSoup

import requests

url = 'https://store-site-backend-static.ak.epicgames.com/freeGamesPromotions?locale=en-US&country=PL&allowCountries=PL'

r = requests.get(url)

data = r.json()

#print(r.text)

for item in data['data']['Catalog']['searchStore']['elements']:
    print(item['title'])
    offers = item['promotions']['promotionalOffers']
    for offer in offers:
        print(offer['promotionalOffers'][0]['startDate'])
        print(offer['promotionalOffers'][0]['endDate'])

Result

Mystery Game
Grand Theft Auto V
2020-05-14T15:00:00.000Z
2020-05-21T15:00:00.000Z

You woould have to digg in data to get other details.

BTW: Maybe you will have to use different values for country and allowCountries

user_18320580
Related