I have few web scrapers written in python to scrape data from ecommerce websites. I want to run these scripts on Azure. What's the best way out? I am looking into Azure functions and Azure Webjobs since I want to have a automated schedules to trigger my scripts. Considering the amount of data to be scraped is huge, will Azure functions will be a costly affair ? And how scalable will be webjobs to get this job done? Are there any other cost effective ways to achieve the task ? (I am New to Azure).
1 Answers
You can try using the Selenium and HTTP Trigger for web scraping.
For Azure function cost on Consumption plan and Premium plan, you can refer to Azure Functions pricing and Reddit/Azure: Azure hosted Web Scraper, good or bad idea?
You can refer to Run Selenium on azure functions, How to create a Selenium web scraper in Azure Functions, Running webscraping Python file with HTML trigger on Azure and Create a Website Scraper for Azure Functions
Related