Cloudflare bypass with Puppeteer

user_18862629 0
Viewed 3102

I'm trying to enter hotbit.io, with my Puppeteer. But I'm met with "Checking your browser before accessing www.hotbit.io" the moment puppeteer tries entering the page. When I run my program in "headless: false" it redirects to the page after 5 seconds. But my problem is, that I want to run it in headless: true. When I run it in headless: true, it timesout on the cloudflare page Screenshot at timeout

I have tried:

It seems like, that cloudflare knows, that I'm having headless activated.

Does anyone know, how I can skip the cloudflare redirecting page?

puppeteer cloudflare node.js
2 Answers

Thank you @BGPHiJACK !

It helped by setting user agent to: Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0

So right after I have init the page, I set user agent.

const page = await browser.newPage();
    
await page.setUserAgent('Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0')
user_188626290

Usually, just using browser-based automation tools such as Puppeteer or Playwright is not enough to bypass the WAF. You’ll want to imitate a real user as much as possible to get ignored by the WAF.

Collecting valid and proper cookies and replicating the original browser properties as much as possible is a must.

In your problem, combining puppeteer-extra-plugin-stealth with random user-agent generation is pretty much enough to bypass the WAF. You can use rand-user-agent to generate random user-agents and pass them on to Puppeteer using setUserAgent.

const puppeteer = require('puppeteer-extra');
const randUserAgent = require('rand-user-agent');
const agent = randUserAgent('desktop');

const StealthPlugin = require('puppeteer-extra-plugin-stealth')();
puppeteer.use(StealthPlugin);

puppeteer.launch({ headless: true }).then(async browser => {
  const page = await browser.newPage();

  await page.setUserAgent(agent);
  await page.goto('https://www.hotbit.io');
  await page.waitForTimeout(5000);
  await page.screenshot({ path: 'test.png', fullPage: true });
  await browser.close();
});

However, this doesn’t imply that every Cloudflare-powered website can be bypassable with that. You can learn more about Cloudflare from here: bypassing Cloudflare.

user_71380910
Related