Scrapyd pause/continue job

Viewed 828

Looks like there is no legal way to pause/continue crawling with Scrapyd, as it was with Scrapy itself (scrapy crawl spider -s JOBDIR=jobdir/spider-1). The only resolution i've found was here:

https://groups.google.com/forum/#!msg/scrapyusers/2VjEBEbOo7s/MSH4GJc2B0sJ

but i haven't understand the idea.

So is it possible to pause/continue job with Scrapyd?

Thanks for your time

1 Answers

The idea is following:

When you run your spider from cmd you can paste additional an argument, for example: scrapy crawl myspider -s JOBDIR=crawls/somespider-1

When you use Scrapyd in endpoint schedule.json you can also paste some settings. If you are using scrapyd-client this will look like this:

s = {'JOBDIR': 'crawls/somespider-1'}
job = api.schedule(project_name, spider_name, settings=s)

After it, you can cancel a job. Next time, when you will run the same job - scrapyd resume previous work

Related