External resources in Puppeteer with Chrome executable fail to load (net::ERR_EMPTY_RESPONSE)

user_2743976 0
Viewed 807

I'm having issues using external resources in a Puppeteer job that I'm running with a full Chrome executable (not the default Chromium). Any help would be massively appreciated!

So for example, if I load a video with a public URL it fails even though it works fine if I hit it manually in the browser.

const videoElement = document.createElement('video');
videoElement.src = src;
videoElement.onloadedmetadata = function() {
  console.log(videoElement.duration);
};

Here's my Puppeteer call:

(async () => {
  const browser = await puppeteer.launch({
    args: [
      '--remote-debugging-port=9222',
      '--autoplay-policy=no-user-gesture-required',
      '--allow-insecure-localhost',
      '--proxy-server=http://localhost:9000',
      '--proxy-bypass-list=""',
      '--no-sandbox', 
      '--disable-setuid-sandbox',
    ],
    executablePath:
      '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
  });

  const page = await browser.newPage();
  logConsole(page);
  
  await page.goto(`http://${hostname}/${path}`, {
    waitUntil: 'networkidle2',
  });
  
  await page.waitForSelector('#job-complete');
  console.log('Job complete!');

  await browser.close();
})();

Unlike many Puppeteer examples, the issue here isn't that my test doesn't wait long enough. The resources fail to load / return empty responses almost instantly.

It also doesn't appear to be an authentication issue - I reach my own server just fine.

Although I'm not running on https here, the URL I try directly in the browser works without SSL.

I should also mention that this is a React (CRA) website and I'm calling Puppeteer with Node.

I can see that at least 3 other external resources (non-video) also fail. Is there a flag or something I should be using that I'm missing? Thanks so much for any help!

puppeteer google-chrome javascript
1 Answers

In my case I had to use puppeteer-extra and puppeteer-extra-plugin-stealth:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

I also found the following flags useful:

const browser = await puppeteer.launch({
    args: [
      '--disable-web-security',
      '--autoplay-policy=no-user-gesture-required',
      '--no-sandbox',
      '--disable-setuid-sandbox',
      '--remote-debugging-port=9222',
      '--allow-insecure-localhost',
    ],
    executablePath:
      '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
  });

Finally, I found it necessary in a few cases to bypass CSP:

await page.setBypassCSP(true);

Please be careful using these rather insecure settings

user_27439760
Related