Using python to collect user information from Instagram

user_11625421 0
Viewed 5225

I am currently using python to collect information from users on instagram, using a text file containing the links of the instagram users. While I can collect the # of followers, # following, and # of posts, I want to be able to collect the bio information from the user. Collecting the bio information will allow me to eventually parse through that information and collect emails. What is the best and simplest way I can do this?

I am not as experienced in Python so I took a sample code from the internet. I tried to analyse the code and use what I know to modify it to my needs but with no results. 

import requests
import urllib.request
import urllib.parse
import urllib.error
from bs4 import BeautifulSoup
import ssl
import json


class Insta_Info_Scraper:

    def getinfo(self, url):
        html = urllib.request.urlopen(url, context=self.ctx).read()
        soup = BeautifulSoup(html, 'html.parser')
        data = soup.find_all('meta', attrs= {'property':'og:description'})
        text = data[0].get('content').split()
        user = '%s %s %s' % (text[-3], text[-2], text[-1])
        followers = text[0]
        following = text[2]
        posts = text[4]
        email = ""
        print ('User:', user)
        print ('Followers:', followers)
        print ('Following:', following)
        print ('Posts:', posts)
        print ('Email:', email)
        print ('---------------------------')

    def main(self):
        self.ctx = ssl.create_default_context()
        self.ctx.check_hostname = False
        self.ctx.verify_mode = ssl.CERT_NONE

        with open('users.txt') as f:
            self.content = f.readlines()
        self.content = [x.strip() for x in self.content]
        for url in self.content:
            self.getinfo(url)


if __name__ == '__main__':
    obj = Insta_Info_Scraper()
    obj.main()

At the moment, I put an empty string as the value for the 'email' variable but want to eventually replace it with code that will get the email from the specific users.

instagram screen-scraping beautifulsoup python
2 Answers

A handy tool for accessing Instagram's public data structures is Instaloader, a Python package that provides both a Python module and a CLI to access Instagram. After doing a pip install instaloader to install it, you can easily get the Profile's metadata saved in JSON files with

instaloader --no-posts --no-profile-pic --no-compress-json profile1 [profile2 ...]

Then you can use jq, "a lightweight and flexible command-line JSON processor", to extract just-saved information, e.g. the following command prints the biography of profile1:

jq -r .node.biography profile1/profile1_*.json

Likewise, a way not leaving Python to access the same information:

import instaloader
L = instaloader.Instaloader()
profile = instaloader.Profile.from_username(L.context, 'profile1')
print(profile.biography)
user_95023820

The best way to do this is using third party libraries like instagram_private_api.

example:

from instagram_web_api import Client

web_api = Client(auto_patch=True, drop_incompat_keys=False)
user_info = web_api.user_info2('instagram')
print(user_info)
user_99773210
Related