I need help to schedule my spiders runs every N minutes. Early I see most people used reactor.callLater and reactor.runfor this purpose but it seems there is no possibilities to use these functionalities now. How can I schedule it programmaticaly?
def run_crawl():
"""
Run a spider within Twisted. Once it completes,
wait 5 seconds and run another spider.
"""
runner = CrawlerRunner(get_project_settings())
runner.crawl(SpiderA)
runner.crawl(SpiderB)
deferred = runner.join()
deferred.addCallback(reactor.callLater, 5, run_crawl)
return deferred
run_crawl()
reactor.run()
At this moment my crawler scheduled by Windows task scheduler, but I want to schedule it programmatically.