How do I web scrape Steam after login?

Question

My goal is to log into Steam's website and populate Excel with my library, and all my friends' libraries to from a table. I can then tabulate & filter what multiplayer games everyone has at the same time, side-by-side.

I've tried the following:

Internet Explorer, but on the login page, I can't click the login button.
Even still, if I manually login and then go to a friend's list of games it's empty.
I think this is an Internet Explorer compatibility issue with Steam's website.
XML but can't get the setRequestHeader to work with login information.
Selenium with Microsoft Edge to login but once it executes clicking the login button the browser closes.
Logging into Steam manually in Microsoft Edge (keeping that window open) then using a macro to get to a friend's game list.
When the new Microsoft Edge window pops up to the page I'm not logged in so it says their game list is private.

Is there a way of web scraping Steam after logging in?

user_1637796 · Answer

I managed to do it with Puppeteer using Node.JS

The trick is to wait until user login in order to continue data extraction. The following is the Script:

const puppeteer = require('puppeteer');
const readline = require('readline');
const fs = require('fs');

// The following function awaits for user input
function askQuestion(query) {
        const rl = readline.createInterface({
                    input: process.stdin,
                    output: process.stdout,
                });

        return new Promise(resolve => rl.question(query, ans => {
                    rl.close();
                    resolve(ans);
                }))
}

(async() => {
    const browser = await puppeteer.launch({
        executablePath: '/usr/bin/chromium',
        headless: false
    }); //Create browser in non-headless mode.
    const page = await browser.newPage(); //Awaits creation of new page
    await page.goto('https://store.steampowered.com/login/');   //Goes to steam login page
    await askQuestion("Login, then press Enter!");  //Now you login and press enter
    await page.goto('https://steamcommunity.com/id/USERNAME/games/?tab=all');   //Loads the page containing the list of all games of the indicated user

    html = await page.evaluate(() => document.querySelector('*').outerHTML);   // Gets the HTML web page
    await fs.writeFile("htmlPage", html, err => {});  //Saves it to a file

    console.log(html);
    await browser.close();   // Closes Puppeteer
} 
)();

This script creates an instance of Puppeteer and opens it in non-headless mode. Then it opens the steam login webpage and waits. At this point, you login manually, go back to console and press Enter. It then opens the page 'https://steamcommunity.com/id/USERNAME/games/?tab=all', and saves the html body to a file called "htmlPage". You can then parse the HTML page however you prefer and get the list of games.

You can extend it if you want it to get the HTML body of several pages by doing a for loop (iterating over a list of id's) after you already managed to log in.

How do I web scrape Steam after login?

1 Answers