KeyError: 'driver' in print(response.request.meta['driver'].title)

user_15190107 0
Viewed 1194

I get the error KeyError:'driver'. I want to create a webcrawler using scrapy-selenium. My code looks like this:

class TestSpider(Spider):
name="test"
start_urls = ["http://crawler-test.com/mobile/dynamic"]

def start_requests(self):
    for url in self.start_urls:
        yield SeleniumRequest(url=url, callback=self.parse)

def parse(self, response):
    print(response.request.meta['driver'].title)

In the settings of the crawler i added:

SELENIUM_DRIVER_NAME = 'chrome'
SELENIUM_DRIVER_EXECUTABLE_PATH = which('chromedriver')
SELENIUM_DRIVER_ARGUMENTS=['--headless']
DOWNLOADER_MIDDLEWARES = {'scrapy_selenium.SeleniumMiddleware': 800}

pretty much everything is from the scrapy-selenium documentation. https://pypi.org/project/scrapy-selenium/

thank you very much for your help.

scrapy selenium python
1 Answers

Answer found from @pcalkins comment

You have two ways to fix this:

Fastest one: Paste your chromedriver.exe file in the same directory that your spider is.

Best one: in SETTINGS.PY put your diver path in SELENIUM_DRIVER_EXECUTABLE_PATH = YOUR PATH HERE

This is you won't use which('chromediver')

user_141049560
Related