I am new to Playwright and trying to create a Scrapy Middleware that uses Playwright to make a request and returns a response.body.
The problem is that I am trying to send headers with my request, but both these methods don't work for me:
browser_context.set_extra_http_headers(headers)
or
page.set_extra_http_headers(headers)
My code looks like this:
async def navigate_page(self, url, headers):
async with async_playwright() as p:
browser = await p.chromium.launch()
context = await browser.new_context()
# At first I tried this method, but it didn't work, so I decided to set headers with page method
# context.set_extra_http_headers(headers)
page = await context.new_page()
await page.set_extra_http_headers(headers)
await page.goto(url)
print(page.request.headers) # showed nothing, debug also showed empty headers
page_content = await page.content()
await browser.close()
return page_content
My headers variable looks like this:
{'referer': 'some url'}
I am pretty sure that I missed something but I searched every article that Google could find and came here in desperation :)