I have a main nodejs application that loops over a list of websites and creates a child process for each website to gather information from that site (scraping).
my problem is that I may have many websites (say 100, 200 or more) what is the best approach for such problem?
is there a better system design for this?
or can I run that much of node js apps on the same device if I skipped using the main app.
please advise.