I am trying to make a Scrapy SitemapSpider and want to have each link that the spider extracts from the sitemap do a get request through the Scraper API (link to documentation: https://www.scraperapi.com/documentation#custom-headers). Do you know how I can do this?
1 Answers
You can set the proxy url in Request's meta attribute.
yield Request(
url=url,
callback=self.parse,
meta={'proxy': 'http://scraperapi:your_key@proxy-server.scraperapi.com:8001'}
)
Then you can simply add Scrapy's HttpProxyMiddleware to your settings.
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
}
You could also implement your own middleware to ensure that every request goes through Scraper API.
class ScraperAPIMiddleware(object):
def process_request(self, request, spider):
request.meta['proxy'] = 'http://scraperapi:your_key@proxy-server.scraperapi.com:8001'
And then enable it in settings.py or your settings object. You'll also need to enable Scrapy's HttpProxyMiddleware to make this approach work, else it won't use the proxy.
DOWNLOADER_MIDDLEWARES = {
'your_project.middleware_file.ScraperAPIMiddleware': 350,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
}
This example uses the simplest implementation of Scraper API's proxy mode. Actually I just finished a more general purpose implementation for my own usage. It's here, feel free to use it.
Related