Run Scrapy Spiders sequential

Question

I have several Scrapy spiders inside my spiders' directory ( let's suppose 50 spiders), now I want to run them sequential (not concurrent)

I could run them concurrent with the following code but because of some policy I've decided to run them sequentially ,

start=datetime.now()
setting = get_project_settings()
process = CrawlerProcess(setting)
for spider_name in process.spiders.list():
    print ("Running spider %s" % (spider_name))
    print(spider_name)
    process.crawl(spider_name) #query dvh is custom argument used in your scrapy

process.start()
print("***********Execution time : {0}".format((datetime.now()-start)))

Also, I tried

for spider_name in process.spiders.list():
   print ("Running spider %s" % (spider_name))
   os.system("pwd ")
   os.system("pwd && scrapy crawl " + spider_name) //pwd to make sure it's in correct path and I see it is

but it seems cant run by os.system, another solution is using a .sh but I'm not sure it's a good idea. I'm looking for a solution to run spiders sequentially?

Run Scrapy Spiders sequential

0 Answers