Unable to get the same requests results from Twitter Web from my local PC and AWS EC2 instance

user_4788695 0
Viewed 207

I want to be able to Scrape Twitter's Trending Topics.

Of course, the natural way to do that, is to use the Twitter API. However, most of the Trends do not come with a Tweet_count, which is key for me.

So I decided to scrape the Twitter website, and it has been a mess.

First, I just went after https://twitter.com/i/trends and it worked fine and still does, on my local computer. Then I tried to set up the script on my AWS EC2 instance, yet I got no results. This is a simplified version of the code:

import requests
from bs4 import BeautifulSoup


url = 'http://twitter.com/i/trends'
r = requests.get(url)
html = r.json()['module_html']
soup = BeautifulSoup(html, 'html.parser')
trends_list = soup.find_all('span', {'class':'u-linkComplex-target trend-name'})
tweet_volume_list = soup.findAll('div', {'class':'js-nav trend-item-stats js-ellipsis'})

and like I said, it worked fine. However, if I run this same code on my Linux server in AWS, the result of r.content is '{}'.

So then I tried going with mobile.twitter.com/i/trends and got a similar problem. I did find with the DevTools on a Private Session that twitter goes to an https://api.twitter.com/2/guide.json endpoint internally, and that is the actual resource that returns the data I'm looking for (trends and tweet volume). However, no matter what I did, with requests, I was unable to access it with python. I tried using the same headers and the same params as the browser, but to no avail.

So then I move to selenium, and just like before, I did get data locally, but not the actual TT data on the server. So at this point I'm pretty lost. I don't know enough web dev to understand exactly if this is a cookie problem or what, nor how to fix it.

TL;DR: I want to scrape Twitter's Trending Topics with python but it's not working.

python-requests web-scraping twitter amazon-ec2 python
1 Answers

The main reason it doesn't work because Twitter blocks AWS EC2 instance IPs. It's not a server problem but a block improvised by Twitter itself. I searched a lot and found the same problems with various libraries used for twitter scraping.

I would recommend using proxies in this case or can change the provider to Linode or digital ocean. I also checked Heroku and turns out its IPs are also blocked after some requests.

Usage of proxies is well-explained in this link of docs from request-docs.

As from your code, the solution should be


import requests
from bs4 import BeautifulSoup

proxies = [
  "http": "http://10.10.1.10:3128",
  "https": "http://10.10.1.10:1080",
  "http": "your proxy server"
]

url = 'http://twitter.com/i/trends'
r = requests.get(url, proxies=proxies)
html = r.json()['module_html']
soup = BeautifulSoup(html, 'html.parser')
trends_list = soup.find_all('span', {'class':'u-linkComplex-target trend-name'})
tweet_volume_list = soup.findAll('div', {'class':'js-nav trend-item-stats js-ellipsis'})

You should also try some of the free proxy servers and python also has libraries like free-proxy that might be of help. Even then I would recommend if the data volume is large then you should consider using multiple proxies and rotate them frequently and also try using async requests library like aiohttp.

user_110986390
Related