Puppeteer returning null on Facebook settings page

user_564514 0
Viewed 434

Update

The problem I am having is reproducible in Chrome without puppeteer. I can see the text in the browser but there is no way to access the data via the javascript console.

Original

facebook privacy settings page has options that are visible from the browser and I can inspect them. The problem is the values from document.querySelector are null.

enter image description here

Here's the url: https://www.facebook.com/settings

Here's the code

const el = await page.$x(
          '/html/body/div[1]/div[3]/div[1]/div/div[2]/div[2]/div[2]/div/ul/li[1]/div/div/ul/li[1]/a/span[3]/div/div[2]'
        );
const v = await page.evaluate((div) => div.textContent, el[0]);

I've also tried by using JS Path

const v = await page.evaluate(
          () =>
            (<HTMLElement>(
              document.querySelector(
                '#u_fetchstream_2_4 > li:nth-child(1) > div > div > ul > li:nth-child(1) > a > span.fbSettingsListItemContent.fcg > div > div._nlm.fwb'
              )
            )).innerText
        );

Always getting the following error:

Error: Evaluation failed: TypeError: Cannot read property 'innerText' of null

I've confirmed that the property is null UNTIL i right-click inspect any value on the page -- bot detection that still shows the page?!

Also tried using xpath in Chrome's console:

$x('//*[text()="Who can see your future posts?"]')

returns []

puppeteer facebook xpath
1 Answers

There is a high chance you are trying to open the URL in different viewport dimensions. This kind of problem happens in responsive websites where the selectors are very different than how you selected them in browser.

The easiest way is to disable the default 800x600 resolution and turn on the headful mode so you can see what is going and and probably why it's failinig.

const browser = await puppeteer.launch({defaultViewport: null, headless: false});

As for unblocking, facebook and other big companies really don't want you to scrape them. So you need to use combination of good proxies, user agent and stay under the radar.

You can try the stealth plugin which will try some unblocking techniques.

const puppeteer = require('puppeteer-extra')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())

Disclaimer: It's highly recommended to use a dummy account for facebook to avoid getting banned. They really does not like bots on their platform.

user_61612650
Related