Capture a screenshot as a table using Puppeteer

Erdem_Ayhan 0
Viewed 42

I am learning to scrape items from a website using Puppeteer. I am using table data from Basketball reference.com to practice. What I have done so far is use the puppeteer to Search the stats of my favorite player (Stephen Curry), access the table page, and take a screenshot of the page which then finishes the scraping process and closes the browser. However, I cannot seem to scrape the table I need and I am completely stuck.

The following is the code I have written so far:

const puppeteer = require("puppeteer");

async function run() {
  const browser = await puppeteer.launch({
    headless: false,
    ignoreHTTPSErrors: true,
  });
  const page = await browser.newPage();
  await page.goto(`https://www.basketball-reference.com/`);
  await page.waitForSelector("input[name=search]");
  await page.$eval("input[name=search]", (el) => (el.value = "Stephen Curry"));
  await page.click('input[type="submit"]');

  await page.waitForSelector(`a[href='${secondPageLink}']`, { visible: true });
  await page.click(`a[href='${secondPageLink}']`);

  await page.waitForSelector();

   await page.screenshot({
    path: `StephenCurryStats.png`,
  });
  await page.close();
  await browser.close();
}
run();

I am trying to scrape the PER GAME table on the following link and take its screenshot. However, I cannot seem to find the right selector to pick and scrape and I am very confused.

The URL is https://www.basketball-reference.com/players/c/curryst01.html

puppeteer javascript
1 Answers

There seems to be at least a couple of issues here. I'm not sure what secondPageLink refers to or the intent behind await page.waitForSelector() (throws TypeError: Cannot read properties of undefined (reading 'startsWith') on my version). I would either select the first search result with .search-item-name a[href] or skip that page entirely by clicking on the first autocompleted name in the search after using page.type(). Even better, you can build the query string URL (e.g. https://www.basketball-reference.com/search/search.fcgi?search=stephen+curry) and navigate to that in your first goto.

The final page loads a video and a ton of Google ad junk. Best to block all requests that aren't relevant to the screenshot.

const puppeteer = require("puppeteer"); // ^16.2.0

let browser;
(async () => {
  browser = await puppeteer.launch({headless: true});
  const [page] = await browser.pages();
  const url = "https://www.basketball-reference.com/";
  await page.setViewport({height: 600, width: 1300});
  await page.setRequestInterception(true);
  const allowed = [
    "https://www.basketball-reference.com",
    "https://cdn.ssref.net"
  ];
  page.on("request", request => {
    if (allowed.some(e => request.url().startsWith(e))) {
      request.continue();
    }
    else {
      request.abort();
    }
  });
  await page.goto(url, {waitUntil: "domcontentloaded"});
  await page.type('input[name="search"]', "Stephen Curry");
  const $ = sel => page.waitForSelector(sel);
  await (await $(".search-results-item")).click();
  await (await $(".adblock")).evaluate(el => el.remove());
  await page.waitForNetworkIdle();
  await page.screenshot({
    path: "StephenCurryStats.png",
    fullPage: true
  });
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close());

If you just want to capture the per game table:

// same boilerplate above this line
await page.goto(url, {waitUntil: "domcontentloaded"});
await page.type('input[name="search"]', "Stephen Curry");
const $ = sel => page.waitForSelector(sel);
await (await $(".search-results-item")).click();
const table = await $("#per_game");
await (await page.$(".scroll_note"))?.click();
await table.screenshot({path: "StephenCurryStats.png"});

But I'd probably want a CSV for maximum ingestion:

await page.goto(url, {waitUntil: "domcontentloaded"});
await page.type('input[name="search"]', "Stephen Curry");
const $ = sel => page.waitForSelector(sel);
await (await $(".search-results-item")).click();
const btn = await page.waitForFunction(() =>
  [...document.querySelectorAll("#all_per_game-playoffs_per_game li button")]
    .find(e => e.textContent.includes("CSV"))
);
await btn.evaluate(el => el.click());
const csv = await (await $("#csv_per_game"))
  .evaluate(el => [...el.childNodes].at(-1).textContent.trim());
const table = csv.split("\n").map(e => e.split(",")); // TODO use proper CSV parser
console.log(table);
ggorlen0
Related