I am using selenium python and was wondering how one effectively scrapes over 50,000 data points in under 24 hours. For example, when I search for products on the webpage 'insight.com' it takes about 3.5 seconds for the scraper to search for the product and grab its price, meaning that with large amounts of data it takes the scraper several days. A part from using threads to simultaneously look up several products at the same time, how else can I speed up this process? I only have one laptop and will have to simultaneously scrape six other similar websites so therefore do not want too many threads and the speed at which the computer operates will slow down significantly. How do people achieve to scrape large amounts of data in such short periods of time?
How to scrape over 50,000 data points from dynamically loading webpage in under 24 hours?
Viewed 192
2 Answers
If you stop using the selenium module, and rather work with a much more sleek and elegant module, like requests, you could get the job done in a matter of mere minutes.
If you manage to reverse engineer the requests being handled, and send them yourself, you could pair this with threading to scrape at some 50 'data points' per second, more or less (depending on some factors, like processing and internet connection speed).
Find an API and use that!!! The goal of both web scraping and APIs is to access web data.
Web scraping allows you to extract data from any website through the use of web scraping software. On the other hand, APIs give you direct access to the data you’d want.
As a result, you might find yourself in a scenario where there might not be an API to access the data you want, or the access to the API might be too limited or expensive.
In these scenarios, web scraping would allow you to access the data as long as it is available on a website.
For example, you could use a web scraper to extract product data information from Amazon since they do not provide an API for you to access this data. However, if you had access to an API, you could grab all the data you want, super, super, super fast!!! It's analogous to doing a query in a database on prem, which is very fast and very efficient, vs. refreshing a webpage, waiting for ALL elements to load, and you can't use the data until all elements have been loaded, and then.....do what you need to do.
Related