Headless Puppeteer - Avoid being detected by Akamai

user_9404560 0
Viewed 3012

Hi I'm trying to scrape a website that is powered by Akamai for bot protection. I'm unable to make it past a Login page due to Akamai blocking my login request.

Firstly I would like to say that, Yes there are a lot of guides on how to avoid being detected by things like Akamai but those are now irrelevant as companies like Akamai are getting better and better through the use of their AI to detect new bots.

So let me tell you the basics of what my script is running:

  • Puppeteer (Headless Mode)
  • puppeteer-extra-plugin-stealth

For the Chrome Flags:

var chromeFlags = [
    '--no-sandbox',
    '--disable-setuid-sandbox',
    '--disable-accelerated-2d-canvas',
    '--no-zygote',
    '--renderer-process-limit=1',
    '--no-first-run',
    '--ignore-certificate-errors',
    '--ignore-certificate-errors-spki-list',
    '--disable-dev-shm-usage',
    '--disable-infobars',
    '--lang=en-US,en',
    '--window-size=1920x1080',
    '--disable-extensions'
  ];

I've also spoofed the Timezones and Viewport:

await page.emulateTimezone("Asia/Singapore");
await page.setViewport({width: (width/2)-21, height: height-111});

Form what I heard, Akamai has specifically known to scrutinize window/screen sizes. I've done everything I think is necessary to ensure the headless mode mimics an actual browser but to no avail.

Theres a website that shows you your browser fingerprints bot.sannysoft.com . I'm currently using that to compare if the headless puppeteer mimics an actual headful browser and so far It seems that it looks like a legitimate browser. Here's the result from that website I got with my headless puppeteer enter image description here

I hope someone is able to tell me if there's anything I should try spoofing next to maybe increase my chance of not being detected by Akamai or point out where I did wrong.

Thanks everyone!

akamai puppeteer headless
2 Answers

I checked the logs and compared them to my own browser and some of my own scripts, it seems completely legitimate. The problem is that Akamai uses different approaches to detect the bots. For example, Akamai is known for using behavioral analysis and AI-powered detection systems, which analyze your activity on the website frequently.

So it's not really enough to mimic a real browser's fingerprint if you can't also bypass the security systems above.

In Akamai-powered websites, the client sends a payload of your behavioral data (click position, potential mouse movement) on the website to the server in an encoded payload. Then the server returns the relevant and valid cookies and headers if the payload passes the WAF check. Since you couldn't get the valid cookies, you're failing the challenge and getting blocked.

For more information about the Akamai and its methods: how does Akamai work?

user_71380910

Try adding the ignoreHTTPSErrors: true when launching puppeteer. Also, check out puppeteer extra and the stealth plugin here:

https://www.npmjs.com/package/puppeteer-extra-plugin-stealth

You can also add me on slack or skype to work on this further!

user_157571080
Related