Scraping Google search result links with Puppeteer

user_14616907 0
Viewed 2441

Below is the code I'm trying to use for web scraping Google when I pass in a specific request but it doesn't return the list of links. I don't understand what's causing it. Can someone help, please?

const puppeteer = require('puppeteer');
const searchGoogle = async (searchQuery) => {
/** by default puppeteer launch method have headless option true*/
const browser = await puppeteer.launch({
headless: false
        });
const page = await browser.newPage();
await page.goto('https://www.google.com/');
await page.type('input[aria-label="Search"]', searchQuery);
await page.keyboard.press('Enter');

/** waitfor while loding the page, otherwise evaulate method will get failed. */
await page.waitFor(5000);
const list = await page.evaluate(() => {
let data = []
/** this can be changed for other website.*/
const list = document.querySelectorAll('.rc .r');
for (const a of list) {
data.push({
'title': a.querySelector('.LC20lb').innerText.trim().replace(/(\r\n|\n|\r)/gm, " "),
'link': a.querySelector('a').href
                })
            }
return data;
        })

await browser.close();

}
module.exports = searchGoogle;
puppeteer node.js javascript
1 Answers

await page.waitFor(5000); in this context causes a race condition. If the page doesn't load in 5 seconds, you can get a false negative. If the page loads quicker than 5 seconds, you've wasted time without reason. Only pick arbitrary delays as a last resort or if it's an intended part of the application logic.

A better approach is to use page.waitForSelector or page.waitForNavigation.

Secondly, I don't see results for the selector .rc .r. I'm not sure how stable Google's CSS selectors are, but .LC20lb seems safe on cursory glance for the time being.

Putting it together gives:

const puppeteer = require("puppeteer"); // ^16.2.0

let browser;
(async () => {
  const searchQuery = "stack overflow";

  browser = await puppeteer.launch();
  const [page] = await browser.pages();
  await page.goto("https://www.google.com/", {waitUntil: "domcontentloaded"});
  await page.waitForSelector('input[aria-label="Search"]', {visible: true});
  await page.type('input[aria-label="Search"]', searchQuery);
  await Promise.all([
    page.waitForNavigation({waitUntil: "domcontentloaded"}),
    page.keyboard.press("Enter"),
  ]);
  await page.waitForSelector(".LC20lb", {visible: true});
  const searchResults = await page.$$eval(".LC20lb", els => 
    els.map(e => ({title: e.innerText, link: e.parentNode.href}))
  );
  console.log(searchResults);
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close())
;

Output (yours may be different depending on what Google shows by the time you run the script):

[
  {
    title: 'Stack Overflow - Where Developers Learn, Share, & Build ...',
    link: 'https://stackoverflow.com/'
  },
  {
    title: 'Stack Overflow - Wikipedia',
    link: 'https://en.wikipedia.org/wiki/Stack_Overflow'
  },
  {
    title: 'Stack Overflow Blog - Essays, opinions, and advice on the act ...',
    link: 'https://stackoverflow.blog/'
  },
  {
    title: 'The Stack Overflow Podcast - Stack Overflow Blog',
    link: 'https://stackoverflow.blog/podcast/'
  },
  {
    title: 'Stack Overflow | LinkedIn',
    link: 'https://www.linkedin.com/company/stack-overflow'
  }
]

Another approach is to encode your search terms as a URL query parameter and navigate directly to https://www.google.com/search?q=your+query+here, saving a navigation and potential selector mishap.

As with many scraping tasks, since the goal is to grab simple hrefs from the document, you can speed up the script substantially by disabling JS, blocking resources or switching to fetch/cheerio and working with the static HTML. For example, the following script runs about 10x faster than Puppeteer:

const cheerio = require("cheerio"); // 1.0.0-rc.12

const query = "stack overflow";
const url = `https://www.google.com/search?q=${encodeURIComponent(query)}`;

fetch(url, { // Node 18 or install node-fetch
  headers: {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36",
  }
})
  .then(res => res.text())
  .then(html => {
    const $ = cheerio.load(html);
    const searchResults = [...$(".yuRUbf a")].map(e => ({
      title: $(e).text().trim(),
      link: e.attribs.href,
    }));
    console.log(searchResults);
  })
;

See also Click an element on first Google search result using Puppeteer.

user_62433520
Related