AWS Lambda + Puppeteer, Navigation Timeout Issue

Viewed 1016

I'm trying to run a puppeteer script to take screenshots from amazon.com, I've used puppeteer-extra, puppeteer-core and chrome-aws-lambda packages which can be found on github and are pretty common I guess.

The script works just fine on my local setup but whenever I try to run it on AWS Lambda, I get a navigation error, sometimes times out, sometimes just returns an error. Here's the code for navigation function, I wrapped it with a while loop so it tries 3 times before giving an error out.

const navigateWithError = (page,url)=>{
    return new Promise(async (res,rej)=>{
      let maxTries = 3
      let i=0
      while(i<maxTries){
        try{
          await page.goto(url, { waitUntil: 'networkidle2' });
          return res(true)
        }catch(err){
          console.log(err)
          console.error('Couldnt navigate, trying again')
          i+=1
        }
      }
      return rej('Couldnt navigate after tries, passing.')
    })
  }

And here is the main function which would go to Amazon.com, do some operations, then take screenshot:

const page = await browser.newPage();
      console.log('created new page')
      await page.setViewport({
        width: 1366,
        height: 768,
      })
      try{
        await navigateWithError(page,'https://www.amazon.com/')
      }catch(err){
        console.error(err)
        res(true)
      }
      await page.evaluate(async ()=>{
        await fetch("https://www.amazon.com/gp/delivery/ajax/address-change.html", {
            "credentials": "include",
            "headers": {
                "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0",
                "Accept": "text/html,*/*",
                "Accept-Language": "tr-TR,tr;q=0.8,en-US;q=0.5,en;q=0.3",
                "X-Requested-With": "XMLHttpRequest",
                "Content-Type": "application/x-www-form-urlencoded;charset=utf-8"
            },
            "referrer": "https://www.amazon.com/",
            "body": "locationType=LOCATION_INPUT&zipCode=75035&storeContext=generic&deviceType=web&pageType=Gateway&actionSource=glow&almBrandId=undefined",
            "method": "POST",
            "mode": "cors"
        });
      })
      console.log('changed postal code')
      try{
        await navigateWithError(page,'https://www.amazon.com/s?k='+query+'&ref=nb_sb_noss_2')
      }catch(err){
        console.error(err)
        res(true)
      }
      console.log('reloaded')
      await page.screenshot({
        path:'/tmp/'+query+'.png',
        type:'png'
      })
0 Answers
Related