NodeJs Pagination,recursive promise problem

Question

I am scraping multiple pages with cheerio and axios in node.js I am having a hard time with Promises, can someone help me return the JSON if I hit the last page? Thanks!

const getWebsiteContent = async (url) => {
    await axios.get(url).then(res => {

        const $ = cheerio.load(res.data)

        pageNum = getTotalpages($);  // Get the pagination

        console.log(url);

        //Some scraping here
    })
    indexPage++; // Increment to the next page

    const nextPageLink = baseUrl + '&page=' + indexPage;      // get next page

    if (indexPage > pageNum) {
        var editedText = text.slice(0, text.length - 1);
        editedText += ']}';
        editedText = JSON.parse(editedText); // I want to return this and use elsewhere
        return editedText;
    }
    setTimeout(async () => {
        getWebsiteContent(nextPageLink); // Call itself
    }, 1000);
}
var myJSON= await getWebsiteContent(baseUrl); // something like this

user_633183 · Answer

I would write getPages as an async generator -

async function* getPages (href, initPage = 0) {
  const res = await axios.get(setPage(href, initPage))
  const $ = cheerio.load(res.data)
  const pages = getTotalpages($)
  yield { page: initPage, dom: $ }
  for (let p = initPage; p < pages; p++) {
    await sleep(1000)
    const r = await axios.get(setPage(href, p))
    yield { page: p, dom: cheerio.load(r.data) }
  }
}

This depends on helper setPage that manipulates the href page number using the url module, which is much safer than hobbling together strings by hand -

function setPage (href, page) {
  const u = new URL(href)
  u.searchParams.set("page", page)
  return u.toString()
}

And another helper, sleep, which prevents the mixing of setTimeout with async-based code. This allows us to easily pause between pages -

async function sleep (ms) {
  return new Promise(r => setTimeout(r, ms))
}

Finally we write scrape which is a simple wrapper around getPages. This allows us to reuse the getPages function to scrape various elements as needed. A benefit of using this approach is that the caller can determine what happens with each page. Below we push to result array, but as another example we could write each page to disk using the fs module. Obviously this for you to decide -

async function scrape (href) {
  const result = []
  for await (const {page, dom} of getPages(href)) {
    console.log("scraped page", page)  // some status message
    result.push(getSomeData(dom))      // get something from each page
  }
  return result
}

scrape(myUrl).then(console.log, console.error)

user_966023 · Answer

You shouldn't be using then with your async / await code. pagination should look something like this:

let response = await axios.get(url)
let $ = cheerio.load(response.data)
// do some scraping
while(url = $('[rel=next]').attr('href')){
  response = await axios.get(url)
  $ = cheerio.load(response.data)
  // do more scraping
}

NodeJs Pagination,recursive promise problem

2 Answers