I am trying to scrape all the reviews from steam games, but steam has both infinite scroll, till it reaches a specific number of reviews, then a button for loading more content. How do I navigate that with scrapy?
My code:
i = 1
url_ = 'https://steamcommunity.com/app/1506830/homecontent/?userreviewscursor=AoIIPwYYan/HzbUD&userreviewsoffset={}0&p={}&workshopitemspage={}&readytouseitemspage={}&mtxitemspage={}&itemspage={}&screenshotspage={}&videospage={}&artpage={}&allguidepage={}&webguidepage={}&integratedguidepage={}&discussionspage={}&numperpage=10&browsefilter=trendweek&browsefilter=trendweek&l=english&appHubSubSection=10&filterLanguage=default&searchText=&maxInappropriateScore=50&forceanon=1'
#url_ is url gathered from inspecting Network while scrolling
class ReviewSpider(scrapy.Spider):
name = 'steam_test'
allowed_domains = ['store.steampowered.com']
start_urls = ['https://steamcommunity.com/app/1517290/reviews/']
def start_requests(self):
yield Request(url, callback=self.parse)
def parse(self, response):
page = get_page(response)
product_id = get_product_id(response)
# Load all reviews on current page.
reviews = response.css('div.apphub_CardContentMain')
for review in (reviews):
yield{
"text":review.css('div.apphub_CardTextContent::text').get(),
"Recommend": review.css('div.title::text').get(),
"date": review.css('div.date_posted::text').get()
}
Pagination section that's not working:
if response.status == 200:
global i
i+= 1
SteamFifaSpider.start_urls = url_.format(i,i,i,i,i,i,i,i,i,i,i,i,i)
How do I fix the infinite scroll for Steam?