Scrapy : restarting a scraping loop when it's finished from within the spider

Viewed 44

I am trying to figure out how to restart a scraping loop once it's over.

My logic is something like:

def start_requests():

    # get a list of coordinates from a database and the last page that got parsed on those coordinates

    for coordinate in coordinates:
        yield scrapy.Request(
            f"{settings.BASE_URL}/search.json?latitude={coordinate.latitude}&longitude={coordinate.longitude}&page={coordinate.current_page}",
            callback = self.parse_coordinates,
            cb_kwargs={'coordinate': coordinate}
        )

def parse_coordinates(self, response, coordinate):
    # parse the page
    # and identify if I reach the last page
    if not last_page: 
        # update coordinate so the scrapy.Request will parse the next page 
        yield scrapy.Request(
            f"{settings.BASE_URL}/search.json?latitude={coordinate.latitude}&longitude={coordinate.longitude}&page={coordinate.current_page}",
            callback = self.parse_coordinates,
            cb_kwargs={'coordinate': coordinate}
            )
    else:
        # reset the page counter to 1 so next time I run the loop I start over

In the end what happens is that I have many coordinates, out of which I'll scrape many pages.

Once this ends, I would like to wait a certain amount of time and reset the loop (i.e. calling again start_requests()). The reset should be performed directly within the spider. The only alternative would be using scrapyd (which I do not know yet) as ultimately I will manage & monitor my jobs using it.

Until now I can't find how to do it. I tried :

  1. Pause the code at the end of start_requests, but it pauses the whole thread, so I do not scrape anything
  2. Uses a recurrent task (task.LoopingCall(self.control_loop)) the same way the Logstats extension is working, but as soon as the main loop finishes, it closes the spider

What would be the correct approach ?

0 Answers
Related