I'm trying to run a puppeteer script to take screenshots from amazon.com, I've used puppeteer-extra, puppeteer-core and chrome-aws-lambda packages which can be found on github and are pretty common I guess.
The script works just fine on my local setup but whenever I try to run it on AWS Lambda, I get a navigation error, sometimes times out, sometimes just returns an error. Here's the code for navigation function, I wrapped it with a while loop so it tries 3 times before giving an error out.
const navigateWithError = (page,url)=>{
return new Promise(async (res,rej)=>{
let maxTries = 3
let i=0
while(i<maxTries){
try{
await page.goto(url, { waitUntil: 'networkidle2' });
return res(true)
}catch(err){
console.log(err)
console.error('Couldnt navigate, trying again')
i+=1
}
}
return rej('Couldnt navigate after tries, passing.')
})
}
And here is the main function which would go to Amazon.com, do some operations, then take screenshot:
const page = await browser.newPage();
console.log('created new page')
await page.setViewport({
width: 1366,
height: 768,
})
try{
await navigateWithError(page,'https://www.amazon.com/')
}catch(err){
console.error(err)
res(true)
}
await page.evaluate(async ()=>{
await fetch("https://www.amazon.com/gp/delivery/ajax/address-change.html", {
"credentials": "include",
"headers": {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0",
"Accept": "text/html,*/*",
"Accept-Language": "tr-TR,tr;q=0.8,en-US;q=0.5,en;q=0.3",
"X-Requested-With": "XMLHttpRequest",
"Content-Type": "application/x-www-form-urlencoded;charset=utf-8"
},
"referrer": "https://www.amazon.com/",
"body": "locationType=LOCATION_INPUT&zipCode=75035&storeContext=generic&deviceType=web&pageType=Gateway&actionSource=glow&almBrandId=undefined",
"method": "POST",
"mode": "cors"
});
})
console.log('changed postal code')
try{
await navigateWithError(page,'https://www.amazon.com/s?k='+query+'&ref=nb_sb_noss_2')
}catch(err){
console.error(err)
res(true)
}
console.log('reloaded')
await page.screenshot({
path:'/tmp/'+query+'.png',
type:'png'
})