Performance with scrapy-playwright seems very poor. For instance I am scraping with the standard request and with playwright 'http://ipinfo.io/ip' where it's hard to have a simplier page.
The requests are simple the only aim is to route the request through playwrite or the standard scrapy downloader.
Playwright
def start_requests(self):
for i in range(0, self.count):
yield scrapy.Request(
"http://ipinfo.io/ip",
callback=self.parse_ipinfo,
dont_filter=True,
meta={
"playwright": True,
},
)
or
Standard
def start_requests(self):
for i in range(0, self.count):
yield scrapy.Request(
"http://ipinfo.io/ip",
callback=self.parse_ipinfo,
dont_filter=True,
)
self.count is set at 10000 though the aim is only to be in a sufficiently long enough loop.
self.parse_ipinfo only logs the response.body
The difference in speed is massive :
- Playwright : 25 pages / min
- Standard : 2200 pages / min
Unfortunately, I am currently forced to go through Playwright (and not even for JS interpretation...) but it's completly killing my scraper's speed.
Is there anything I can do to improve it ?
*** first findings ***
Playing with the REACTOR_THREADPOOL_MAXSIZE can help through increasing the parallelism. By default the REACTOR_THREADPOOL_MAXSIZE is 10, increasing it to 100 allowed me to reach ~90 pages / min.