scrapy : parallel AND sequential running of spider

user_15749434 0
Viewed 178

I have a scrapy project with multiple spiders. Some take minutes, some take hours, and anything in between - however that elapsed time is usually about the same for each run - so you can assume that scraper X runs for about the same time as scrapers Y and Z together.

What I want to do is instead of running the all in parallel starting at T0, I want to start scrapers 1,2,3 at the start, then chain scrapers 4,5,6 after 2 finishes, and 7,8,9 after 3 finishes to smooth out the downstream processing requirements (concurrent database connections etc)

I think I need to chain the deferreds, and there are some clear examples in the docs, but i'm not sure how to set that up as well as some running in parallel - current starting code is below (each spider is in its own external file)

from scrapy.utils.project import get_project_settings
from scrapy.crawler import CrawlerProcess

setting = get_project_settings()

process = CrawlerProcess(setting)
process.crawl('scraper1')
process.crawl('scraper2')
process.crawl('scraper3')
...etc...
...etc...
process.start()
scrapy twisted python
1 Answers

Figured out the answer - easier than I thought.

There's no need to worry about stopping the reactor (hence why it is commented out)

Scraper 1 and scraper 2 start at the same time, scraper 3 starts after scraper 2 finishes.


from scrapy.utils.project import get_project_settings
from scrapy.crawler import CrawlerProcess

setting = get_project_settings()

process = CrawlerProcess(setting)

@defer.inlineCallbacks
def crawl_seq():
    global process

    yield process.crawl('scraper2')
    yield process.crawl('scraper3')
    #reactor.stop()

crawl_seq()
process.crawl('scraper1')

process.start()
user_157494340
Related