I can't get body when request from Bloomberg url

user_7371219 0
Viewed 1186

I want to get the body from a Bloomberg url.

This is the code I am using:

   const options = {
    url: 'https://www.bloomberg.com/quote/CCMP:IND',
    headers: {
      'User-Agent': 'request'
    }
  };

  function callback(error, response, body) {
    if (!error && response.statusCode == 200) {
      console.log(body);
    }
  }

  request(options, callback);

But I am getting this message when I console log the body:

<!doctype html>
<html>
<head>
    <title>Bloomberg - Are you a robot?</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">

I have found a similar problem and the solution using BeautifulSoup in Python... but I cannot find the solution in NodeJs: How to scrape text from a <p> elements "id"

node.js web-scraping javascript
1 Answers

Firstly, you can try to add all the headers that a normal browser is having when doing a request with request, just like this:

headers: {
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'en-US,en;q=0.9,fr;q=0.8,ro;q=0.7,ru;q=0.6,la;q=0.5,pt;q=0.4,de;q=0.3',
    'cache-control': 'max-age=0',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'
}

If this is still not working then you should try Puppeteer, it is a Chromium based browser API made by google and you can simulate the exact workflow of a browser with NodeJs.

Here is a good example on how to use it and start off: NodeJs Scraping with Puppeteer

user_26834670
Related