Disclaimer: I have all permissions to scrape data from websites' owners.
I am BI analyst, and currently work on NLP project. I gather and analyze public opinion about certain products using online data. I have multiple (34, one for each website participated in the project) web crawlers (web scraping whatever it is called).
The cycle of data gathering starts automatically within windows scheduler. Every Monday windows scheduler runs “.bat” file, which runs 34 (each of web crawlers) python scripts simultaneously. Sometimes errors appear and stop code execution, so I run “cycle” manually again.
Full cycle of each crawlers includes:
- Data gathering
- Data processing
- Some ML\DL models (classifying and text summarization)
- Exports to DB
- Reports generation (using R and markdown)
I want to build infrastructure around web crawlers to make data gathering more efficient and professional. For instance, I want to virtualize each of web crawlers within Docker to avoid errors that may interrupt the entire cycle of gathering. I am completely new to containerization and MLops, so I do not know how it works in “production”. I want to control workflow and make it clear and sustainable especially for further modifications\extensions.
I am looking for books, articles, and other tutorials, as well as advices on how to build and organize such web scraping eco-system from data gathering and reports generation.