Requests returns a status code of 429 for URL https://www.instagram.com/google

user_16781043 0
Viewed 72

I'm trying to code an Instagram-webscraper in Python to return values like a person's followers, the number of posts etc. Let's just take Google's Instagram-account for this example.

Here is my code:

import requests
from bs4 import BeautifulSoup

link = requests.get("https://www.instagram.com/google")
soup = BeautifulSoup(link.text, "html.parser")

print(soup)
print(link.status_code)

Pretty straightforward. However, if I run the code, it prints link.status_code = 429. It should be 200, for any other website it prints 200. Also, when it prints soup, it doesnt show what I actually want. Not the HTML for the account is shown, but the HTML for the Instagram-Error-page.

Why does requests open the instagram error page, not the account from the link provided?

python-requests web-scraping beautifulsoup python
1 Answers

To get correct response from the server, set User-Agent HTTP header:

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"
}

link = requests.get("https://www.instagram.com/google", headers=headers)
soup = BeautifulSoup(link.text, "lxml")

print(link.status_code)
print(soup.select_one('meta[name="description"]')["content"])

Prints:

200
12.5m Followers, 33 Following, 1,642 Posts - See Instagram photos and videos from Google (@google)
user_100359850
Related