how find which element cause error in promis.map and log it in console, or skip it by code?

Admin 1
Viewed 157

I using following code, but some URL from URL List, get 500 Error because of structure of the page.

I get the error exactly on .map((htmlOnePage, index) => line when some URLs page not valid, and the flow control of program goes at Catch Part. How I can find which URL is invalid?

const requestPromise = require('request-promise');
const Promise = require('bluebird');
const cheerio = require('cheerio');

for (var i = 1; i <= 250; i++) {
  p = "https://mywebsite.com/" + i.toString()

  urls[i - 1] = p
}

Promise.map(urls, requestPromise)
  .map((htmlOnePage, index) => {
    const $ = cheerio.load(htmlOnePage);

    $('.txtSearch1').each(function() {
      var h = "";
      h = $(this).text()
      h = h.replace(/(\r\n|\n|\r)/gm, "")

      html44.push(h)
    })
    
    shareTuple[urls[index]] = html44;
    html44 = []

    fs.writeFileSync("data.json", JSON.stringify(shareTuple))
  }, {
    concurrency: 1
  })
  .then()
  .catch((e) => console.log('We encountered an error' + e));

in other word how I can show which URL get into catch?

cheerio node.js web-scraping javascript
2 Answers

You should add try catch phrases inside all iterating functions to pin point the problem. and do the logging for every one based on op that they are catching.
For example i would wrap it like this:

try {
      const $ = cheerio.load(htmlOnePage);

    $('.txtSearch1').each(function () { 
        try {
        var h="";
        h=$(this).text()
        h= h.replace(/(\r\n|\n|\r)/gm, "")

        html44.push (h)
        }catch (error) {
             console.log('Error in getting p text');      
             console.log(error);
        }
} catch (error) {
   console.log('Error in loading'+ htmlOnePage);
   console.log(error);
}

You can break down the error more by looking up through error object values to find the problem if you want to manualy remove it.
The other way to programaticaly remove it is to try doing the request for the page after creating it and also wrapping it in try catch. If it throws an exception you can add in catch to remove it from the list.
That text before console log of error is just so you can see where it broke.

Admin1

Use a wrapper around requestPromise that catches the error. Note, the return undefined is not really needed. It's just for clarification, that in case of an error nothing is returned.

const requestPromise = require('request-promise');
....

const noThrowRequest = async (url) => {
  try {
    return await requestPromise(url);
  } catch (e) {
    return undefined;
  }
}

Or if you prefer .then().catch() you can do it as follows

const noThrowRequest = (url) => {
  return requestPromise(url)
    .then(result => { return result; })
    .catch(e => { return undefined; });
}

And then use that wrapper instead of requestPromise and check whether the current result valid or not. I don't know what you want to do in case of an invalid result, so I just return from the callback without any further ado. Adapt that if necessary.

Promise.map(urls, noThrowRequest)
  .map((htmlOnePage, index) => {
    if (htmlOnePage === undefined) {
      console.log(`there was an error at index ${index}`);
      return; //stop callback for erronous indexes.
    }

    ...
  }
user_37769270
Related