How to set class variable through __init__ in Python?

Viewed 181

I am trying to change setting from command line while starting a scrapy crawler (Python 3.7). Therefore I am adding a init method, but I could not figure out how to change the class varible "delay" from within the init method.

Example minimal:

class testSpider(CrawlSpider):

    custom_settings = {
        'DOWNLOAD_DELAY': 10,  # default value
    }

    """ get arguments passed over CLI
        scrapyd usage: -d arg1=val1
        scrapy  usage: -a arg1=val1
    """
    def __init__(self, *args, **kwargs):
        super(testSpider, self).__init__(*args, **kwargs)

        self.delay = kwargs.get('delay')

        if self.delay:
            testSpider.custom_settings['DOWNLOAD_DELAY'] = self.delay
            print('init:', testSpider.custom_settings['DOWNLOAD_DELAY'])

    print(custom_settings['DOWNLOAD_DELAY'])

This will not change the setting unfortunatelly:

scrapy crawl test -a delay=5
10
init: 5

How can the class variable be changed?

1 Answers

I am trying to change setting from command line while starting a scrapy crawler (Python 3.7). Therefore I am adding a init method...
...
scrapy crawl test -a delay=5

  1. According to scrapy docs. (Settings/Command line options section) it is requred to use -s parameter to update setting
    scrapy crawl test -s DOWNLOAD_DELAY=5

  2. It is not possible to update settings during runtime in spider code from init or other methods (details in related discussion on github Update spider settings during runtime #4196

Related