Run Multiple Spider sequentially

user_4149986 0
Viewed 4847
Class Myspider1
#do something....

Class Myspider2
#do something...

The above is the architecture of my spider.py file. and i am trying to run the Myspider1 first and then run the Myspider2 multiples times depend on some conditions. How Could I do that??? any tips? 

configure_logging()
runner = CrawlerRunner()
def crawl():
    yield runner.crawl(Myspider1,arg.....)
    yield runner.crawl(Myspider2,arg.....)
crawl()
reactor.run()

I am trying to use this way.but have no idea how to run it. Should I run the cmd on the cmd(what commands?) or just run the python file??

thanks a lot!!!

scrapy-spider scrapy web-crawler python
2 Answers

You need to use the Deferred object returned by process.crawl(), which allows you to add a callback when the crawl is finished.

Here is my code

def start_sequentially(process: CrawlerProcess, crawlers: list):
    print('start crawler {}'.format(crawlers[0].__name__))
    deferred = process.crawl(crawlers[0])
    if len(crawlers) > 1:
        deferred.addCallback(lambda _: start_sequentially(process, crawlers[1:]))

def main():
    crawlers = [Crawler1, Crawler2]
    process = CrawlerProcess(settings=get_project_settings())
    start_sequentially(process, crawlers)
    process.start()
user_46719530
Related