How to make puppeteer wait for page redirect from Cloudflare browser check?

user_11402733 0
Viewed 4510

I am scraping a website where after submitting a form I get redirected to this -

Checking your browser before accessing <Website Name>.
This process is automatic. Your browser will redirect to your requested content shortly.

Please allow up to 5 seconds…

DDoS protection by Cloudflare
Ray ID: <Some ID>

Now, usually when I myself submit that form manually from the "real web browser" I get redirected to the main contents almost immediately after this browser checking stuff appears. But in puppeteer, it doesn't.

I have tried to use page.waitForNavigation() but I couldn't get it to work. Is there any way to actually get past this checking process? Or puppeteer is just getting blocked?

Thanks in advance!

puppeteer cloudflare ddos browser javascript
3 Answers

You can use puppeteer-extra which enables you to add different plugins like StealthPlugin or AdblockerPlugin. Have a look at their documentation.

user_157363870

You can wait for specific element on the web page. For example header or something.

Using await page.waitForElement(selector);

user_109854830

Cloudflare's WAF checks a few things, such as the request's headers and TLS handshake fields while you're trying to connect to the website. These are collected in payloads and sent to Cloudflare’s servers. An inconsistency will get you blocked or redirected.

An easier way to bypass this protection is by using puppeteer-extra, which lets you use plugins in Puppeteer. Then you can use puppeteer-extra-plugin-stealth, which implements many features to conceal the client and get ignored by the WAF.

If the problem is with your IP (or geolocation), you can check out the proxy-chain package.

Here's a minimal working example with using puppeteer-extra-plugin-stealth and proxy-chain (with HTTP proxy):

const puppeteer = require('puppeteer-extra')
const randUserAgent = require('rand-user-agent');
const proxyChain = require('proxy-chain');
const StealthPlugin = require('puppeteer-extra-plugin-stealth')();

puppeteer.use(StealthPlugin)

const agent = randUserAgent("desktop");
const proxyUrl = 'http://userid:pw@ip:port'
const proxy = async () => {
  return proxyChain.anonymizeProxy(proxyUrl);
}

puppeteer.launch({ headless: true,
  args: [
    '--proxy-server='+proxy
  ]
}).then(async browser => {
  const page = await browser.newPage()
  
  await page.setUserAgent(agent);
  await page.goto('https://bot.sannysoft.com', { waitUntil: 'networkidle2' })
  await page.waitForTimeout(5000) 
  await page.screenshot({ path: 'test.png', fullPage: true })
  await browser.close()
})

You can learn more about the methods Cloudflare is using from here: Cloudflare's bot protection.

user_71380910
Related