Im using scrapy to crawl a news website on a daily basis. How do i restrict scrapy from scraping already scraped URLs. Also is there any clear documentation or examples on SgmlLinkExtractor.
Scrapy - how to identify already scraped urls
Viewed 9769
5 Answers
For today (2019), this post is the best answer for this problem.
https://blog.scrapinghub.com/2016/07/20/scrapy-tips-from-the-pros-july-2016
It's a lib to handle MIDDLEWARES automatcally.
Hope to help someone. I've spent a lot of time seaching for this.
Related