Let's say that I have a queue of mixed web pages with mixed domains that needs to be downloaded. I want to trigger the downloader only after specific time has elapsed, let's say 1 minute. Meanwhile I don't want my downloader to stay idle but I want it to download different pages from different domains. How can I prevent downloading from the same domain more than one time per minute and at the same time keep using my downloader for other downloads? How should I keep the time and the domains? Are there any best practices for this?
0 Answers
Related