Fix no referer even after setting in middlewares process_request in Scrapy

Question

I have this code here inside middlewares.py > process_request:

    def process_request(self, request, spider):
        refreerr_list = [
            'https://google.com',
            'https://amazon.com',
            'https://facebook.com',.....
        ]
        referr = random.choice(refreerr_list)

        request.header['referer'] = referr

As it should show referer but in the log, it shows:

2022-07-17 07:17:13 [scrapy.core.engine] DEBUG: Crawled (200)  (referer: None)

Do I have to add something in settings.py?

But it works when doing in settings.py:

DEFAULT_REQUEST_HEADERS = {
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',,
  'referer': 'https://www.amazon.com/',
}

Thanks for any help!!

user_17829451 · Answer

You need to uncomment this section in your settings.

DOWNLOADER_MIDDLEWARES = {
    'spiders.middlewares.MyProjectDownloaderMiddleware': 543,
}

The name of the class will depend on the name of your project.

Fix no referer even after setting in middlewares > process_request in Scrapy

1 Answers