Why can i not scrape twitter with beautiful soup in python?

user_6737387 0
Viewed 1649

I wonder why can i not scrape twitter profiles with python beautiful soup.

from bs4 import BeautifulSoup
import requests
import urllib3

url = "https://twitter.com/Microsoft"
r = requests.get(url)
bs = BeautifulSoup(r.content, 'lxml')
match = bs.find_all('span')

match contains a single element containing the following text

Something went wrong, but don’t fret — let’s give it another shot.

I can't fathom why is this happening, is it because twitter stopped allowing web scrapers to scrape it's content ? Any solution would be appreciated .

web-scraping twitter beautifulsoup python
1 Answers

Some sites are using techniques to prevent web scraping. However there are many technical solutions to scraping Twitter data, if you use these you are likely to be contravening the Twitter Terms of Service which states:

“…scraping the Services without the prior consent of Twitter is expressly prohibited”

And that you should not “…access or search or attempt to access or search the Services by any means (automated or otherwise) other than through our currently available, published interfaces ”

You can access Twitter data through Twitter’s APIs, but depending on your usage Twitter may charge for this.

user_117450140
Related