scrapy close reason in pipeline

Viewed 1503

I use scrapy pipelines to save items in db and it works fine for this purpose. Now I'm try to write in db a record with spider closed reason. When I raise CloseSpider exception I can yield specific item before raise (and save record to db). But if spider closed with "finished" I can't do same thing. What I tried:

  1. make connect spider_closed signal to spider_closed method inside spider method and yield item from there - that is not working
  2. inside spider_closed set some spider variable to reason value and then try to read it in pipeline's close_spider method. The problem here is that pipeline's close_spider called before spider's spider_closed. And also there is no reason param in pipeline's close_spider signature: close_spider(self, spider)
  3. Working solution which I have now - direct execute query inside spider's spider_closed. But I want to avoid that and separate my db logic from spider logic

So, can anyone provide me how to access spider close reason inside pipeline?

UPD: Thank's to Paul I register signal handler inside pipeline from_crawler method (almost like inside spider) and it works for me

@classmethod
def from_crawler(cls, crawler):
    temp = cls()
    crawler.signals.connect(temp.spider_closed, signal=scrapy.signals.spider_closed)
    return temp

def spider_closed(self, reason):
    logging.info("REASON: {}".format(reason))
1 Answers
Related