Scrapy how to set DEPTH_LIMIT from a command line argument

Viewed 1315

I currently have DEPTH_LIMIT set in the settings module for the scraper I am building. I would like to be able to pass a depth limit in as a command line argument. I have tried the following as a constructor for the crawler (and variations of it):

    def __init__(self, max_depth=3, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
        self.settings['DEPTH_LIMIT'] = int(max_depth)

but, I get an error and the stack dump ends with:

  File "/usr/local/lib/python2.7/dist-packages/scrapy/spider.py", line 41, in crawler
      assert hasattr(self, '_crawler'), "Spider not bounded to any crawler"
   AssertionError: Spider not bounded to any crawler

Even trying to print self.settings['DEPTH_LIMIT'] in the constructor causes and error. How can I set the DEPTH_LIMIT in a crawler from a command line argument?

Thanks!

1 Answers
Related