I am trying to change setting from command line while starting a scrapy crawler (Python 3.7). Therefore I am adding a init method, but I could not figure out how to change the class varible "delay" from within the init method.
Example minimal:
class testSpider(CrawlSpider):
custom_settings = {
'DOWNLOAD_DELAY': 10, # default value
}
""" get arguments passed over CLI
scrapyd usage: -d arg1=val1
scrapy usage: -a arg1=val1
"""
def __init__(self, *args, **kwargs):
super(testSpider, self).__init__(*args, **kwargs)
self.delay = kwargs.get('delay')
if self.delay:
testSpider.custom_settings['DOWNLOAD_DELAY'] = self.delay
print('init:', testSpider.custom_settings['DOWNLOAD_DELAY'])
print(custom_settings['DOWNLOAD_DELAY'])
This will not change the setting unfortunatelly:
scrapy crawl test -a delay=5
10
init: 5
How can the class variable be changed?