How to avoid being banned while scraping data from a login based site?

user_13354003 0
Viewed 1965

I'm trying to create a script using which I can parse few fields from a website without getting blocked. The site I wish to get data from requires credentials to access it's content. If it were not for login thing, I could have bypassed the rate limit using rotation of proxies.

As I'm scraping content from a login based site, I'm trying to figure out any way to avoid being banned by that site while scraping data from there. To be specific, my script currently can fetch content from that site flawlessly but my ip address gets banned along the way if I keep on scraping.

I've written so far (consider the following site address to be a placeholder):

import requests
from bs4 import BeautifulSoup

url = "https://stackoverflow.com/users/login?ssrc=head&returnurl=https%3a%2f%2fstackoverflow.com%2f"

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
    req = s.get(url)

    payload = {
        "fkey": BeautifulSoup(req.text,"lxml").select_one("[name='fkey']")["value"],
        "email": "some email",
        "password": "some password",
    }
    
    res = s.post(url,data=payload)
    soup = BeautifulSoup(res.text,"lxml")
    for post_title in soup.select(".summary > h3 > a.question-hyperlink"):
        print(post_title.text)

How can I avoid being banned while scraping data from a login based site?

python-requests web-scraping beautifulsoup python-3.x python
6 Answers

Going direct to the point of "any efficient way to avoid being banned" there is no way.

I would compare this situation with a shark attack. It was the shark's decision, not yours.

However, there are some techniques that you can use to mitigate the "shark attack"... But first, let's make it clear that you are "attacking" the shark first, swimming in its domain.

The technique would be: "Creating a human scraping script".

The word human here is referred to make random mistakes sometimes. Some of them listed below:

  • Insert some random delays between your tasks;
  • Click on some wrong link, wait few seconds, go back;
  • Log you out from the system, wait a minute or two, log you in again;
  • If you have a list of links on a page to click and then grab the data for each page, don't do it in order;
  • If you have a page that shows the results in pages, get the results don't do it in order (ex. 1, 5, 2, 9, 10, 3, 7, 4, 8, 6)
  • Don't rush, get few data each day

However, the most effective way would be to contact the website owner and offer a partnership or pay for accessing the data using an API or something like that if they have this service.

user_98937250

I add a similar issue writing a library using selenium to fetch financial data from websites.

The only efficient way I found was to add a random delay around 1 second to mimic a human behavior, with the following command after each request (at the end of my loop)

time.sleep( 1.0 +  numpy.random.uniform(0,1) )

You will have around 60 requests by minutes. You can adjust the 1.0 to have a frequency which is ok for the website.

If the website computes the frequency of your requests, it will finds exactly one minute, which is higly suspicious, but right now I didn't find one which did this.

user_67219300

While trying to scrape websites that do not want to be scraped is not recommended, it is possible. You would need to setup proxies for your code to run through. I have used ScrapingHub to host my my scraping projects. It will change its ip address through proxies automatically.

user_148655850

one more useful thing to do is to switch to selenium. Selenium spawns a web browser instance every time you run your scraper so the site will think you are a legit person.

user_103724520

In addition to @Paulo's answer that discusses many techniques to act human you can workaround the IP blocking issue by frequently (or after some requests) changing the IP address from which you are sending the request.

Proxy Method:

It can be done programmatically. This website shares some points to take care when doing so. A related question is asked at SO.

Existing Tools

There are some existing tools/framework that might help. Scrapy, Tor. The tutorial here provides amazing techniques that you can use as well. You can rotate user-agent, proxy, and referer to differentiate your request every time.

To prevent the credentials from being banned, either rotate your credentials or contact them directly. If they are asking credentials then it is the idea that they don't want to load that part of their website. Act human would be the preferable way in that case.

user_36943810

Use the selenium web driver to open a browser and load the page you want, and then you can do the scrape on the browser you have running. This is a way that the site does not block you since it contains headers etc. that simulates the interaction of a user with the site

for example, using chrome web

self.driver = webdriver.Chrome(executable_path=path +/driver//chromedriver.exe', chrome_options=self.options) self.driver.get('site to scrape')

this will open a browser, then scrape over this browser check this selenium-python

user_56376110
Related