I am attempting to scrape the result of a reverse image searching using Google Images.
My start URL is https://www.google.com/searchbyimage?image_url=DIRECT_IMAGE_URL, which works as intended in the browser.
However, when I attempt to scrape it via Scrapy, the request is redirected 3 times, as shown;
with the resulting webpage looking like
instead of the results page. Here's my Scrapy settings.py
SPLASH_URL = 'http://splash:8050'
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPIDER_MIDDLEWARES = {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
REDIRECT_MAX_TIMES = 2
HTTPCACHE_ENABLED = True
I have considered;
- Lack of Javascript - so I used Splash to render the page
- Cookies - I think it's working?
- Setting maximum redirects - Didn't work
Here's my call to SplashRequest;
yield SplashRequest(url, self.parse, meta={
'cookiejar': i,
'wait': 0.5,
'splash': {
'args': {
'html': 1,
'png': 1,
},
'splash_headers': {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11'
} # optional; a dict with headers sent to Splash
}
Any idea what's causing the 3 redirects? The first redirect is necessary for the result, but the 2nd and 3rd ones aren't and give me the wrong page.

