Unable to click on <a> tag using Puppeteer

user_15530206 0
Viewed 47

I am trying to scrape a real estate website. This is my first time using Puppeteer. Up to now I have managed to open the link and click on the "accept all" button for cookies. Now I am trying to click on the "sign in" button. This is my code:

const puppeteer = require("puppeteer");

const link1 =
  "https://www.daft.ie/for-rent/walled-garden-wyckham-way-dundrum-dublin-14/3988400";

(async () => {
  const browser = await puppeteer.launch({
    headless: false,
    defaultViewport: null,
  });
  const page = await browser.newPage();
  await page.goto(link1);

  const elements = await page.$x("//button[normalize-space()='Accept All']");
  await elements[0].click();
  console.log("cookies accepted");

  const signin = await page.$x("//a[contains(., 'Sign in')]");
  await signin[0].click();

  console.log("Sign in clicked");

  await browser.close();
})();

This is the error I get after I run the script:

 throw new Error('Execution context was destroyed, most likely because of a navigation.');
              ^

Error: Execution context was destroyed, most likely because of a navigation.
    at rewriteError (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:280:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async ExecutionContext._ExecutionContext_evaluate (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:226:56)
    at async ElementHandle.evaluateHandle (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/JSHandle.js:97:16)
    at async Object.internalHandler.queryAll (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/QueryHandler.js:63:30)
    at async ElementHandle.$$ (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/ElementHandle.js:116:17)
    at async /Users/alex/Desktop/coding/daft/index.js:18:1

I verified the xPath I am using for the a tag sign in button is correct.

Here's the HTML of the sign in link:

<a data-testid="top-level-active-nav-link" href="/auth/authenticate" class="Navigationstyled__TopLevelItemActiveLink-sc-1tagpb7-6 bsFoce">Sign in</a>

What am I doing wrong here?

Link of the target page - list

puppeteer javascript
1 Answers

page.click() issues a trusted event that sometimes fails due to visibility issues. A workaround is using element.evaluate(el => el.click()) to trigger a native DOM click.

Secondly, you'll want to waitForNavigation using Promise.all when triggering an event that initiates a navigation.

Putting it together:

const puppeteer = require("puppeteer"); // ^16.2.0

let browser;
(async () => {
  browser = await puppeteer.launch({headless: false});
  const [page] = await browser.pages();
  const url =
    "https://www.daft.ie/for-rent/walled-garden-wyckham-way-dundrum-dublin-14/3988400";
  await page.goto(url, {waitUntil: "domcontentloaded"});
  const xp = "//button[normalize-space()='Accept All']";
  const cookiesBtn = await page.waitForXPath(xp);
  await cookiesBtn.click();
  const signIn = await page.waitForXPath("//a[contains(., 'Sign in')]");
  await Promise.all([
    signIn.evaluate(el => el.click()),
    page.waitForNavigation()
  ]);
  await page.waitForSelector("#username");
  await page.type("#username", "hello");
  await page.type("#password", "world");
  await page.click('input[name="login"]');
  await page.waitForTimeout(100000); // pause just to show it's working
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close())
;

Note that the sign-in link is https://www.daft.ie/auth/authenticate, so you may be able to achieve your goals more reliably and efficiently if you use this as your root URL.

Another trick I use for scripts with complex sign-in workflows that involve a lot of frames, oauth and other difficult-to-automate components is to do it all by hand in Puppeteer's browser while saving the cookies and site data in the userDataDir. It seems that this website should be workable without that trick but it's worth mentioning in case you run into trouble.

user_62433520
Related