I am working on a custom RSS feed generator, which scrapes a site and stores the scraped info in a JSON file, and then builds an RSS feed from the JSON. This would be run daily from a background process.
The site displays articles linearly, meaning the newer articles are before the older ones, and have a smaller page number.
Assuming the past data never changes, is there a way to only scrap new articles(i.e. stop when the rest data exist in the JSON)?
My current idea is
- Load the JSON within the spider and check if the scraped item is already in it, but I am uncertain if the scrapping process is ordered.
- Save the checkpoint manually such as the id of the most recent article, but I would prefer a more general solution that does not depend on data having a unique id.
Also, please let me know if there is a better way to implement this pipeline.