Request-promise not working for this URL?

Mister X 0
Viewed 24

I Just tried the below code for get response for this URL - https://www.data.ai/account/login/?_ref=header

but it didn't gets success and not return any results(code went to standby for more hours). Whether the problem is with URL or my code?

var rp = require('request-promise');
rp.get('https://www.data.ai/account/login/?_ref=header',{ resolveWithFullResponse: true });

if i tried some other url such as google.com then it gets results.

request-promise javascript
1 Answers

Your code does not work because JavaScript is used to build a DOM on the site. Or the site is using anti-scraping protection. I would suggest you to use Puppeteer:

const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");

puppeteer.use(StealthPlugin());

async function scrapeDataAI() {
  const BASE_URL =
    "https://www.data.ai/account/login/?_ref=header";

  const browser = await puppeteer.launch({
    headless: false,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });
  const page = await browser.newPage();
  await page.goto(BASE_URL);
  // More commands here...
  /* You may view the docs at:
       https://pptr.dev/
     And more magic at:
       https://www.npmjs.com/package/puppeteer
     Github:
       https://github.com/puppeteer/puppeteer
  */

  // await browser.close();
}

scrapeDataAI();
Ritik Banger0
Related