Local scraper and cloud scraper returns different results

Question

I tried to scrape instagram and getting different results. Function was deploing by command below. When I tried very simillar function from this video: https://www.youtube.com/watch?v=i8THvr03FaY&list=WL&index=54&t=0s everything works fine. It seems to be problem with instagram imo, but maybe somebody have idea how fix this.

My scraping snippet:

const puppeteer = require('puppeteer');

let browserPromise = puppeteer.launch({
  args: ['--no-sandbox'],
});

exports.igprof = async (req, res) => {
  const username = req.query.username || 'instagram';
  let url = 'https://www.instagram.com/' + username;

  const browser = await browserPromise;
  const context = await browser.createIncognitoBrowserContext();
  const page = await context.newPage();

  await page.goto(url, { waitUntil: 'networkidle2' });
  const html = await page.evaluate(() => document.body.innerText);
  //   const json = JSON.parse(html);

  //   if (json.graphql) {
  //     console.log(json.graphql.user.profile_pic_url_hd);
  //     await page.goto(json.graphql.user.profile_pic_url_hd);
  //   }

  const image = await page.screenshot({
    clip: {
      x: 240,
      y: 140,
      width: 320,
      height: 320,
    },
  });

  //   res.setHeader('Content-Type', 'image/png');
  res.setHeader('Content-Type', 'text/html');
  //   res.send(image);
  res.send(html);

  context.close();
};

package.json

{
  "name": "ig-prof",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "functions-framework --target=igprof"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@google-cloud/functions-framework": "^1.2.1",
    "puppeteer": "^1.19.1"
  }
}

deploy command:

gcloud functions deploy igprof --trigger-http --runtime=nodejs10 --memory=1024mb --region=europe-west1

results localhost:

instagram Verified Follow 6,462 posts 362m followers 59 following Instagram #ShareBlackStories about.instagram.com/blog/announcements/introducing-instagram-reels-announcement Made Us  Guides  Reels ⭐️ #TryThisAtHome SBS  Pride 2020  Juneteenth Self-Care Tips 2020 Vision POSTS GUIDES IGTV TAGGED Show More Posts from instagram Related Accounts See All virat.kohli Verified Virat Kohli Follow billieeilish Verified BILLIE EILISH Follow championsleague Verified UEFA Champions League Follow marshmellomusic Verified marshmello Follow carryminati Verified   Follow sachintendulkar Verified Sachin Tendulkar Follow tiktok Verified TikTok Follow nickiminaj Verified Barbie Follow leonardodicaprio Verified Leonardo DiCaprio Follow samsungwithgalaxy Verified Samsung #withGalaxy Follow sony Verified Sony Follow To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Instagram through cookies. Learn more, including about available controls: Cookies Policy. Close Search Log In to Instagram Log in to see photos and videos from friends and discover other accounts you'll love. Log In Sign Up Log In Sign Up ABOUTHELPPRESSAPIJOBSPRIVACYTERMSLOCATIONSTOP ACCOUNTSSUGGESTED ACCOUNTSHASHTAGSLANGUAGE Afrikaans Čeština Dansk Deutsch Ελληνικά English Español (España) Español Suomi Français Bahasa Indonesia Italiano 日本語 한국어 Bahasa Melayu Norsk Nederlands Polski Português (Brasil) Português (Portugal) Русский Svenska ภาษาไทย Filipino Türkçe 中文(简体) 中文(台灣) বাংলা ગુજરાતી हिन्दी Hrvatski Magyar ಕನ್ನಡ മലയാളം मराठी नेपाली ਪੰਜਾਬੀ සිංහල Slovenčina தமிழ் తెలుగు Tiếng Việt 中文(香港) Български Français (Canada) Română Српски Українська © 2020 INSTAGRAM FROM FACEBOOK

results google cloud

Instagram Phone number, username, or email Password Log In OR Log in with Facebook Forgot password? Don't have an account? Sign up Get the app. ABOUTHELPPRESSAPIJOBSPRIVACYTERMSLOCATIONSTOP ACCOUNTSHASHTAGSLANGUAGE Afrikaans Čeština Dansk Deutsch Ελληνικά English Español (España) Español Suomi Français Bahasa Indonesia Italiano 日本語 한국어 Bahasa Melayu Norsk Nederlands Polski Português (Brasil) Português (Portugal) Русский Svenska ภาษาไทย Filipino Türkçe 中文(简体) 中文(台灣) বাংলা ગુજરાતી हिन्दी Hrvatski Magyar ಕನ್ನಡ മലയാളം मराठी नेपाली ਪੰਜਾਬੀ සිංහල Slovenčina தமிழ் తెలుగు Tiếng Việt 中文(香港) Български Français (Canada) Română Српски Українська © 2020 INSTAGRAM FROM FACEBOOK

why this returns different results and how I can fix that?

user_6272061 · Answer

Instagram redirects you to the login page instead of showing the public profile, because most likely the scraper was detected and instagram prevents you from scraping their data on purpose.

Although it works locally, when you deploy a scraper to the cloud the environment from which the scraper is run changes. And IP addresses from cloud services are often flagged, meaning instagram can tell that you run puppeteer from a backend deployed on a google cloud, and decide to block those requests.

Options you can try:

use puppeteer stealth plugin. While this is a potential solution for other websites, for scraping instagram it does not work.
test different cloud providers (google cloud, AWS, heroku, but ideally chose smaller more unknown services)
use a proxy service like brightdata.com. As far as I know they support instagram scraping.

Local scraper and cloud scraper returns different results

1 Answers