web scraping : iterate through every page of web app using puppeteer

user_13286790 0
Viewed 505

I'm a complete beginner in javascript and puppeteer and I am trying to get the first 151 pokemon descriptions from the bulbapedia website https://en.wikipedia.org/wiki/List_of_generation_I_Pok%C3%A9mon In the picture below I copy the XPath of the blue marked element of a single pokemon instance and it is the text I want to show

enter image description here

Using my code below I can succeed in grabbing the element and showing the text in a json value but I can only do it manually for one pokemon at a time . What I want is to use puppeteer to iterate through each page and do this for the first 151 pokemon
My code :

const puppeteer = require('puppeteer');

async function getDesc(url){

    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto(url);
    //xpath of the selected text above in the pic 
    const [el] = await page.$x('//*[@id="mw-content-text"]/table[5]/tbody/tr[1]/td/table/tbody/tr[2]/td/table/tbody/tr[1]/td');
    const text = await el.getProperty('textContent');
    srcTxt = await text.jsonValue();

    console.log({srcTxt});

    browser.close();

}



//give url for a specific pokemon as input 
getDesc('https://bulbapedia.bulbagarden.net/wiki/Bulbasaur_(Pok%C3%A9mon)');

I believe that a for loop that iterates through each url for every pokemon instance is the solution . However I do not know how to implement this using puppeteer and I would appreciate your help . Thank you in advance .

puppeteer web-scraping javascript
1 Answers

If you have a URL array, you can try something like this:

'use strict';

const puppeteer = require('puppeteer');

(async function main() {
  try {
    const browser = await puppeteer.launch();
    const [page] = await browser.pages();

    const urls = ['https://example.org/1', 'https://example.org/2'];
    const allData = [];

    for (const url of urls) {
      await page.goto(url);

      const data = await page.evaluate(() => {
        return document.querySelector('a').innerText;
      });

      allData.push(data);
    }

    console.log(allData);

    await browser.close();
  } catch (err) {
    console.error(err);
  }
})();
user_26255600
Related