I am trying to figure out how to restart a scraping loop once it's over.
My logic is something like:
def start_requests():
# get a list of coordinates from a database and the last page that got parsed on those coordinates
for coordinate in coordinates:
yield scrapy.Request(
f"{settings.BASE_URL}/search.json?latitude={coordinate.latitude}&longitude={coordinate.longitude}&page={coordinate.current_page}",
callback = self.parse_coordinates,
cb_kwargs={'coordinate': coordinate}
)
def parse_coordinates(self, response, coordinate):
# parse the page
# and identify if I reach the last page
if not last_page:
# update coordinate so the scrapy.Request will parse the next page
yield scrapy.Request(
f"{settings.BASE_URL}/search.json?latitude={coordinate.latitude}&longitude={coordinate.longitude}&page={coordinate.current_page}",
callback = self.parse_coordinates,
cb_kwargs={'coordinate': coordinate}
)
else:
# reset the page counter to 1 so next time I run the loop I start over
In the end what happens is that I have many coordinates, out of which I'll scrape many pages.
Once this ends, I would like to wait a certain amount of time and reset the loop (i.e. calling again start_requests()). The reset should be performed directly within the spider. The only alternative would be using scrapyd (which I do not know yet) as ultimately I will manage & monitor my jobs using it.
Until now I can't find how to do it. I tried :
- Pause the code at the end of start_requests, but it pauses the whole thread, so I do not scrape anything
- Uses a recurrent task (task.LoopingCall(self.control_loop)) the same way the Logstats extension is working, but as soon as the main loop finishes, it closes the spider
What would be the correct approach ?