Can't scrape from a page I navigate to by using Puppeteer

user_14819659 0
Viewed 380

I'm fairly new to Puppeteer and I'm trying to practice keep tracking of a selected item from Amazon. However, I'm facing a problem when I try to retrieve some results from the page.

The way I intended this automation to work is by following these steps:

  • New tab.
  • Go to the home page of Amazon.
  • Enter the given product name in the search element.
  • Press the enter key.
  • Return the product title and price.

Check this example below:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: false,
  });
  const page = await browser.newPage();
  await page.setRequestInterception(true);


  page.on('request', (req) => {      // don't load any fonts or images on my requests. To Boost the  performance

    if (req.resourceType() == 'font' /* || req.resourceType() == 'image' || req.resourceType() == 'stylesheet'*/) {
      req.abort();
    }
    else {
      req.continue(); {

      }
    }
  });

const baseDomain = 'https://www.amazon.com';

  await page.goto(`${baseDomain}/`, { waitUntil: "networkidle0" });

await page.click("#twotabsearchtextbox" ,{delay: 50})

  await page.type("#twotabsearchtextbox", "Bose QuietComfort 35 II",{delay: 50});
  await page.keyboard.press("Enter");
  await page.waitForNavigation({
    waitUntil: 'networkidle2',
  });

  let productTitle = await page.$$(".a-size-medium, .a-color-base, .a-text-normal")[43]; //varible that holds the title of the product

  console.log(productTitle );

  debugger;

})();

when I execute this code, I get in the console.log a value of undefined for the variable productTitle. I had a lot of trouble with scraping information from a page I navigate to. I used to do page.evaluate() and it only worked when I'm scraping from the page that I have told the browser to go to.

puppeteer node.js javascript
1 Answers

The first problem is on this line:

let productTitle = await page.$$(".a-size-medium, .a-color-base, .a-text-normal")[43];
// is equivalent to:
let productTitle = await (somePromise[43]);

// As you guessed it, a Promise does not have a property `43`,
// so I think you meant to do this instead:
let productTitle = (await page.$$(".a-size-medium, .a-color-base, .a-text-normal"))[43];

Once this is fixed, you don't get the title text, but a handle to the DOM element. So you can do:

let titleElem = (await page.$$(".a-size-medium, .a-color-base, .a-text-normal"))[43];
let productTitle = await titleElem.evaluate(node => node.innerText);

console.log(productTitle); // "Microphone"

However, I'm not sure that simply selecting the 43rd element will always get you the one you want, but if it isn't, that would be a topic for another question.

user_19137290
Related