Apify How can I stop enqueueLinks function from opening new page in browser

user_19342685 0
Viewed 165

I am trying to get links from a page and then navigate to the next page through a click of a button. The issue is I first need to add all the links on the first page to a queue however the enqueueList function seems to start a new page, causing a "Node not found error" when I try to click on an element. Any advice would be helpful!

exports.handleList = async ({ request, page }, requestQueue) => {
    await Apify.utils.enqueueLinks({
        page: page,
        requestQueue: requestQueue,
        selector: "#traziPoduzeca > tbody > tr > td > span > a",
        baseUrl: "https://www.fininfo.hr/Poduzece/[.*]",
        transformRequestFunction: (request) => {
            request.userData = {
                label: "DETAIL",
            };
            return request;
        },
    });

    log.info('about to scrap urls')

    
    await page.waitForTimeout(120);
    
    let btn_selector =
        "//div[@class='contentNav'][1]//div[@class='pagination'][1]//span[@class='current']/following-sibling::a";
    let buttonEle = await page.$x(btn_selector);

    log.info(`length of pagination is ${buttonEle.length}`);

    for (let index = 0; index < buttonEle.length; index++) {
        await page.waitForTimeout(120);

        await buttonEle[index].click();

        log.info("clicked");

        await page.waitForTimeout(200);

        await Apify.utils.enqueueLinks({
            page: page,
            requestQueue: requestQueue,
            selector: "#traziPoduzeca > tbody > tr > td > span > a",
            baseUrl: "https://www.fininfo.hr/Poduzece/[.*]",
            transformRequestFunction: (request) => {
                request.userData = {
                    label: "DETAIL",
                };
                return request;
            },
        });
    }
};
apify puppeteer web-scraping
1 Answers

It's not the enqueueLinks function that is causing the error, is the navigation that is happening. You need to await for the navigation if you want to stay on the same page, something like this:

const btn_selector =
        "//div[@class='contentNav'][1]//div[@class='pagination'][1]//span[@class='current']/following-sibling::a";

// log.info(`length of pagination is ${buttonEle.length}`);

for (;;) {
  // the button handle will change everytime the navigation happens
  // so you need to re-evaluate every time
  const [buttonEle] = await page.$x(btn_selector);
   
  if (!buttonEle) {
    log.info("no pagination element found");
    break;
  }
  
  await page.waitForTimeout(120);

  await Promise.all([
     page.waitForNavigation(), // this will not destroy the context of current page
     buttonEle.click(), 
  ]);

  log.info("clicked");

  await page.waitForTimeout(200);

  // enqueue links...
}

user_6473800
Related