Scraping Google Images with Scrapy / Splash - Getting redirected

Viewed 1163

I am attempting to scrape the result of a reverse image searching using Google Images.

My start URL is https://www.google.com/searchbyimage?image_url=DIRECT_IMAGE_URL, which works as intended in the browser.

However, when I attempt to scrape it via Scrapy, the request is redirected 3 times, as shown;

Image reverse search redirects

with the resulting webpage looking like

Google search result

instead of the results page. Here's my Scrapy settings.py

    SPLASH_URL = 'http://splash:8050'
    DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
     SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
      }
     DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
      HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
      REDIRECT_MAX_TIMES = 2
      HTTPCACHE_ENABLED = True 

I have considered;

  1. Lack of Javascript - so I used Splash to render the page
  2. Cookies - I think it's working?
  3. Setting maximum redirects - Didn't work

Here's my call to SplashRequest;

yield SplashRequest(url, self.parse, meta={
        'cookiejar': i,
        'wait': 0.5,
        'splash': {
          'args': {
              'html': 1,
              'png': 1,
          },
          'splash_headers': {
            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11'
          }       # optional; a dict with headers sent to Splash
      }

Any idea what's causing the 3 redirects? The first redirect is necessary for the result, but the 2nd and 3rd ones aren't and give me the wrong page.

1 Answers
Related