Multiple Spiders Within External Python File - How to run subsequently

user_13534073 0
Viewed 41

Does anyone know how I can run multiple spiders subsequently within a Python file that calls Scrapy at some stage? At the moment my code is breaking because I think the next spider is being blocked because the first one is running.

if __name__ == "__main__":
    c = CrawlerProcess({
        'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36',
        'FEED_FORMAT': 'json',
        'FEED_URI': 'spider_one.json'
    })
    c.crawl(SpiderOneSpider)
    c.start() #the script will block here until the crawling is finished according to the documentation

    c = CrawlerProcess({
        'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36',
        'FEED_FORMAT': 'json',
        'FEED_URI': 'spider_two.json'
    })
    c.crawl(SpiderTwoSpider)
    c.start()

Also not sure if there is a neater way to write this code? I'm basically using SpiderTwo to do the same thing SpiderOne does but just exports a different JSON file.

scrapy python-3.x
1 Answers

Remove the start() line on the middle, once the crawler starts you won't be able to add any more spider to the crawl.

if __name__ == "__main__":
    c = CrawlerProcess({
        'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36',
        'FEED_FORMAT': 'json',
        'FEED_URI': 'spider_one.json'
    })
    c.crawl(SpiderOneSpider)

    c = CrawlerProcess({
        'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36',
        'FEED_FORMAT': 'json',
        'FEED_URI': 'spider_two.json'
    })
    c.crawl(SpiderTwoSpider)
    c.start()

If you actually need to run them sequentially I suggest you take a look here: https://docs.scrapy.org/en/latest/topics/practices.html#running-multiple-spiders-in-the-same-process

user_32006900
Related