I currently have DEPTH_LIMIT set in the settings module for the scraper I am building. I would like to be able to pass a depth limit in as a command line argument. I have tried the following as a constructor for the crawler (and variations of it):
def __init__(self, max_depth=3, *args, **kwargs):
super(MySpider, self).__init__(*args, **kwargs)
self.settings['DEPTH_LIMIT'] = int(max_depth)
but, I get an error and the stack dump ends with:
File "/usr/local/lib/python2.7/dist-packages/scrapy/spider.py", line 41, in crawler
assert hasattr(self, '_crawler'), "Spider not bounded to any crawler"
AssertionError: Spider not bounded to any crawler
Even trying to print self.settings['DEPTH_LIMIT'] in the constructor causes and error. How can I set the DEPTH_LIMIT in a crawler from a command line argument?
Thanks!