The installed Reactor does not match the requested one

Question

I'm trying to run through the examples on scrape-playwrights docs in running the scrolling example on quotes.toscrape.com/scroll and I can't even get to scraping because of a issue with the reactors:

URL SPIDER TEST
***********************
SCRAPE STARTED
***********************
2022-08-11 15:47:38 [scrapy.crawler] INFO: Overridden settings: {'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'} crawled: >

The code is:

import csv
import json
import pygsheets
import scrapy
from scrapy_playwright.page import PageMethod
import json
from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings
from scrapy.utils.reactor import install_reactor
from scrapy.crawler import CrawlerProcess
from scrapy.crawler import CrawlerRunner
import datetime as dt
from datetime import date
from twisted.internet import reactor, defer
import tempfile

def breaker(comment):
    print('***********************')
    print(comment)
    print('***********************')

class UrlSpider(scrapy.Spider):
    name = "Url"

    custom_settings={
        'DOWNLOAD_HANDLERS':{
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
    }

    def start_requests(self):
        yield scrapy.Request(
            url='http://quotes.toscrape.com/scroll',
            meta=dict(
                playwright=True,
                playwright_include_page=True,
                playwright_page_methods=[
                    PageMethod('wait_for_selector','div.quote'),
                    PageMethod('evaluate','window.scrollBy(0, document.body.scrollHeight)'),
                    PageMethod('wait_for_selector','div.quote:nth-child(11)'),
                ],
            ),
        )
async def parse(self, response):
    page=response.meta['playwright_page']
    await page.screenshot(path='quotes.png',full_page=True)
    await page.close()
    return {'quotes_count':len(response.css('div.quote'))}

print('URL SPIDER TEST')

configure_logging()
settings=get_project_settings()
runner = CrawlerRunner(settings)

@defer.inlineCallbacks
def crawl():
    breaker('SCRAPE STARTED')
    bug=runner.crawl(UrlSpider)
    reactor.close()
    yield bug

url_list=crawl()
print('crawled: '+str(url_list))
reactor.run()

I've been trying to find the solution for hours with no luck, and I'm using CrawlerRunner because I want to automate the code at some point, but even with CrawlerProcess I get the error.

I'm also using custom settings because I've had issues with project setting not being added with get_project_settings, custom settings let's me make sure it's used.

If I remove the setting for twisted reactor in custom settings the spider scraps and yields but the reactor error happens again and it doesn't retrieve anything.

The installed Reactor does not match the requested one

0 Answers