I use scrapy pipelines to save items in db and it works fine for this purpose. Now I'm try to write in db a record with spider closed reason. When I raise CloseSpider exception I can yield specific item before raise (and save record to db). But if spider closed with "finished" I can't do same thing. What I tried:
- make connect spider_closed signal to spider_closed method inside spider method and yield item from there - that is not working
- inside spider_closed set some spider variable to reason value and then try to read it in pipeline's close_spider method. The problem here is that pipeline's close_spider called before spider's spider_closed. And also there is no reason param in pipeline's close_spider signature: close_spider(self, spider)
- Working solution which I have now - direct execute query inside spider's spider_closed. But I want to avoid that and separate my db logic from spider logic
So, can anyone provide me how to access spider close reason inside pipeline?
UPD: Thank's to Paul I register signal handler inside pipeline from_crawler method (almost like inside spider) and it works for me
@classmethod
def from_crawler(cls, crawler):
temp = cls()
crawler.signals.connect(temp.spider_closed, signal=scrapy.signals.spider_closed)
return temp
def spider_closed(self, reason):
logging.info("REASON: {}".format(reason))