Is it good to use Heroku as a proxy?

user_17123424 0
Viewed 273

Heroku provides dynamic ip addresses which change when the dyno is restarted. So, instead of paying for proxies, can take advantage of this Heroku feature.

I want to do large-scale automated web scrapping without being blocked.

There already exists a Heroku app for doing exactly the above-mentioned process. But I cannot remember the name (something like this nameOfApp.herokuapp.com). To use it we have to do something like https://nameOfApp.herokuapp.com/destination.com. I came across it in a Stack Overflow question.

I predict that the domain is anywhere.herokuapp.com but cannot use it.

node.js web-scraping heroku ip proxy
1 Answers

Heroku provides dynamic ip addresses which change when the dyno is restarted. So, instead of paying for proxies, can take advantage of this Heroku feature.

I want to do large-scale automated web scrapping without being blocked

Heroku is unlikely to help in this regard.

For many use cases, there's never a valid reason for requests to come from cloud services. As a result, it is common for sites that don't want to get scraped to block entire IP ranges, e.g. ones for Amazon Web Services (the underlying cloud infrastructure that Heroku is built upon).

Many users find that their scrapers work fine locally, even for extended periods of time, and immediately fail when deployed to Heroku.

Could it help? Sure. But this will be highly dependent on the site you're trying to scrape, the mitigations in place at the time, and many other factors.

Side note that I'm sure you'll just ignore: please respect sites' terms of service. If they don't want you scraping them, don't scrape them. If they implement technical barriers, that's a pretty good sign that they don't want to be scraped.

user_3545770
Related