Fetching HTML information after page is loaded

user_14722763 0
Viewed 73

So basically I'm trying to make a discord bot to fetch the player data within a game called Venge.io. Their api is private and requires a token to access.

I tried using the code below to fetch the overall page of my specific stats:

    if(message.content.startsWith(`${prefix}test`)){
        JSDOM.fromURL("https://social.venge.io/#Izzi").then(dom => {
            console.log(dom.serialize());
        });
    }

The outcome resulting in things such as this:

<li>
   <span class="label">KILLS</span>
   <span class="value">{{details.kills}}</span>
</li>

(Snippet of the long amount of html)

When inspecting it loaded manually, it gives a result such as this:

<li>
   <span class="label">KILLS</span>
   <span class="value">11748</span>
</li>

I'm looking to figure out how I can fetch the loaded html instead of the raw html so I can then use this for multiple users and more things available.

jsdom discord fetch html javascript
1 Answers

I've just checked the site you mentioned and it's a Vue.js SPA. You can't use JSDOM for parsing dynamic web pages; you will need a headless browser, like Puppeteer, which can execute JS functions.

If you open the page, you can see the Vue application (a global app variable). In your browser's console you can access app.details and see that it contains all the data you may need, including details.kills.

So you can spin up a headless browser and grab the variables you need, without touching the DOM. Check out the snippet below, it should work fine:

const puppeteer = require('puppeteer');

async function getUserDetails(username) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  const url = `https://social.venge.io/#${username}`;

  await page.goto(url, { waitUntil: 'networkidle0' });

  const handle = await page.evaluateHandle(() => app.details);
  const details = await handle.jsonValue();

  await browser.close();

  return details;
}

getUserDetails('Izzi')
  .then((details) => console.log(details.kills));
user_61263730
Related