Iterate over table using puppeteer

Question

I want to get data from a website which is in table. First i try to get the whole table and then get the tr's and td's that are inside it. The code i have now just return empty array.

const puppeteer = require("puppeteer");

async function run() {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();
  await page.goto(
    "https://www.basketball-reference.com/leagues/NBA_2021_standings.html" //Eastern Conference
  );

  var temp = [];
  const data = await page.evaluate(() => {
    const tableBody = document.querySelector(
      'table[id="confs_standings_E"] tbody'
    );

     for (var i = 0; i < tableBody.length; i++) {
      const tr = tableBody[i].querySelectorAll("tr");
      for (var j = 0; j < tr.length; j++) {
       const td = tr[j].querySelectorAll("td").innerText;
       temp.push(td);
  }
}
  });

  console.log(temp);
  //await browser.close();
}

run();

UPDATE

I tried the solution that was posted and it worked so thanks for that but i wanted to try to get solution in another way. The code below gets the right number of elements, but they are all undefined This is what i get in console:

0: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
1: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
2: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
3: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
4: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
5: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
6: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
7: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
8: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
9: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
10: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
11: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
12: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
13: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]
14: (7) [undefined, undefined, undefined, undefined, undefined, undefined, undefined]

And here is new code:

async function run() {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();
  await page.goto(
    "https://www.basketball-reference.com/leagues/NBA_2021_standings.html" //Eastern Conference
  );

  seznamEkip = [];
  const ekipe = await page.$$("#confs_standings_E tbody tr");

  for (const ekipa of ekipe) {
    const podatki = await ekipa.$$("td");
    const spread = [...podatki].map((element) => element.innerText);
    seznamEkip.push(spread);
  }

  console.log(seznamEkip);
}

run();

Thanks for any help

user_1108513 · Answer

Recall that document.querySelector returns a single element, not an array. Therefore, iterating over its length with a for-loop does not make sense, since it does not have a length -- it's just an element. You may find it useful to iterate over the element's .children instead.

As an alternative, consider utilizing querySelectorAll and utilizing a slightly different selector instead. For instance, this will select every td in the tbody, and return an array of their innerTexts:

const data = await page.evaluate(() => {
    const tableBody = document.querySelectorAll('#confs_standings_E tbody td');
    return Array.from(tableBody).map(element => element.innerText);
});
console.log(data);

Iterate over table using puppeteer

1 Answers