Puppeteer page.click works, but page.evaluate + document click doesn't work

user_2235936 0
Viewed 2426

I've seen so many posts (for example, see here and here), saying that I could click on something via the following code,

await page.click('.route-redirect-box');   // via Puppeteer page.click

await page.evaluate((css_selector) => {
  document.querySelector(css_selector).click();  // or via page.evaluate
}, css);

However, as I tested on some websites, looks like page.click always works, but page.evaluate doesn't, using headless: false mode.

For example, this website page, I tried to click on something as follows,

var css = '#searchPaginationTop > nav > a:nth-child(5)';
await page.evaluate((css_selector) => { document.querySelector(css_selector).click();}, css);

nothing happened at all, but if I use page.click, it worked as expected.

I'm thinking that, the element I wanted to click is not a normal clickable element, since the html code for that element is as follows,

<a class="svg" data-goto-page="3" data-total-pages="3" data-ga="event" data-ga-category="Brands at allbeauty-Burberry-Pagination" data-ga-action="Brands at allbeauty-Burberry-Pagination-Next-Touch" data-ga-label="Brands at allbeauty-Burberry-Pagination-Next-Link">
    <svg viewBox="0 0 21.9 38.7" alt="Next Page" title="Next Page ">
        <use xlink:href="#icon-ab-arrow-right">
        </use>
    </svg>
</a>

Could it be because this element is some data-ga stuff, so page.evaluate cannot click on it?

puppeteer javascript
2 Answers

Short answer

  • page.evaluate(() => document.querySelector('SELECTOR').click()); just fires the click event
  • page.click('SELECTOR') tries to mimic human behavior when clicking

Explanation

Let's check the documentation for both methods to really understand what is happening.

page.evaluate(() => document.querySelector('SELECTOR').click());

Let's see what the MDN documentation says:

[...] It fires the element's click event.

That's all it does. It just fires the click event, so that all handlers listening to that click event of the element are called. This means, it does not care if the element is outside of the current viewport. The element might even be hidden (via CSS) and the click event would still fire.

Let's compare that to the "puppeteer-way":

page.click

The part of the puppeteer documentation regarding page.click:

This method fetches an element with selector, scrolls it into view if needed, and then uses page.mouse to click in the center of the element. [...]

That means, that puppeteer mimics human behavior here. First, the element is scrolled into view, then the mouse is moved on top of the element (triggering any other events like mouseover, mouseenter, etc. on the way). Finally, the button is clicked by simulation a mouse (see the corresponding Mouse class in the puppeteer code). This also triggers any related events (like mousedown).

Complex UI libraries might not like it, when you trigger JavaScript events on your own. Keep in mind, they are often optimized for human interaction, not for interaction with bots. That means, the UI library might listen to the mousedown or mouseenter event (as an example) instead of directly listening to the click event.

Behaving "human-like"

When interacting with an unknown website, it is best to try to behave as human as possible. Even pages that do not have any specific "anti-bot" measures, might use frameworks that expect a specific flow of events.

By the way, you are not the only one having this problem. Check out these questions for similar problems:

user_56275990

From what I can tell, the content seem to be injected dynamically. That means, by using waitForSelector of Puppeteer, you can wait for it to happen before continuing. Something along these lines should do (can't replicate it myself, since I don't know know how to trigger the update of the nav):

await page.click('.route-redirect-box');
const css = '#searchPaginationTop > nav > a:nth-child(5)';
await page.waitForSelector(css);
await page.evaluate((css_selector) => { document.querySelector(css_selector).click();}, css);
user_51899200
Related