What are the necessary cookies to set so the "Before you continue to Google" page won't appear?

mavdee 0
Viewed 33

I'm scraping data from Google using Puppeteer. But before Puppeteer gets to the google page, an annoying popup appears (screenshot).

I want to prevent this from happening so that I don't have to click the "reject" / "allow" button in Puppeteer every time. What are the necessary cookies to achieve it?


  const browser = await puppeteer.launch()
  const page = await browser.newPage()
  await page.goto("https://google.com/")

      const cookies = [
    {
      domain: '.google.com',
      expirationDate: 9999999999.298648,
      name: 'CONSENT',
      ...
    },
  ]

  await page.setCookie(...cookies)
gdprconsentform google-chrome cookies puppeteer
1 Answers

Instead of trying to hard code the cookies, just save them.

  • Come up with a file path where the cookies file will be saved, e.g. cookiesFilePath.
  • After launching puppeteer, check to see if a file exists at the path. If it exists load it.
  • Keep your code for navigating the consent form, as you will use it at least once. At the end of the accepting/rejecting, save page.cookies to cookiesFilePath
const cookiesPath = './cookies.json';
const browser = await puppeteer.launch()
const page = await browser.newPage()

if(await fs.existsSync(cookiesPath)){
  let cookies = require(cookiesPath)
  for(let cookie of cookies){
    await page.setCookie(cookie)
  }
}

await page.goto("https://google.com/")

const hasConsentForm = async ()=>{
  // create query that will return consent form
  const consentFormQuerySelector=''
  // I dont remember if page.$eval querySelector fails 
  // throws errors or not
  try{
    return await page.$eval(consentFormQuerySelector,el=>Boolean(el))
  }catch(err){
    console.log(err)
    return false
  }
}

const navigateConsentForm = async ()=>{
  // add your existing code that navigates the consent form
  
  // ...

  // after the consent form is cleared, save cookies
  let cookies = await page.cookies()
  fs.writeFileSync(cookiesPath,JSON.stringify(cookies,null,2))
}

if(await hasConsentForm()){
 await navigateConsentForm()
}

Assuming that the lack of cookies or the lack of a certain cookie property is the reason that the consent form pops up, this is will make it so that after running once, the form doesnt show again.

PhantomSpooks0
Related