Scrapy display response.request.url inside zip()

user_17734768 0
Viewed 39

I'm trying to create a simple Scrapy function which will loop through a set of standard URLs and pull their Alexa Rank. The output I want is just two columns: One showing the scraped Alexa Rank, and one showing the URL which was scraped.

Everything seems to be working except that I cannot get the scraped URL to display correctly in my output. My code currently is:

import scrapy

class AlexarSpider(scrapy.Spider):
    name = 'AlexaR'
    #Will update allowed domains and start URL once I fix this problem
    start_urls = ['http://www.alexa.com/siteinfo/google.com/', 
    'https://www.alexa.com/siteinfo/reddit.com']

    def parse(self, response):
        rank = response.css(".rankmini-rank::text").extract()
        url_raw = response.request.url
    
        #extract content into rows
        for item in zip(url_raw,rank):
            scraped_info = {
                str('url_raw') : item[0],
                'rank' : item[1]
            }

        yield scraped_info

And then when run, the code outputs a table showing:

AlexaRank Output

url_raw rank
h
t 21
t
h
t 1
t

These are the correct scraped rankings (21 and 1) but the url_raw field is showing "h" or "t", rather than the actual URL string value. I've tried converting the url_raw variable to a string with no luck.

How can I set the variable up such that it displays the correct URL?

Thank you in advance for any help!

scrapy python
1 Answers

Here zip() takes 'rank' which is a list and 'url_raw' which is a string so you get a character from 'url_raw' for each iteration.

Solution with cycle:

import scrapy
from itertools import cycle


class AlexarSpider(scrapy.Spider):
    name = 'AlexaR'
    #Will update allowed domains and start URL once I fix this problem
    start_urls = ['http://www.alexa.com/siteinfo/google.com/',
                  'https://www.alexa.com/siteinfo/reddit.com']

    def parse(self, response):
        rank = response.css(".rankmini-rank::text").extract()
        url_raw = response.request.url
        #extract content into rows
        for item in zip(cycle([url_raw]), rank):
            scraped_info = {
                str('url_raw'): item[0],
                'rank': item[1]
            }
            yield scraped_info

Solution with list:

import scrapy


class AlexarSpider(scrapy.Spider):
    name = 'AlexaR'
    #Will update allowed domains and start URL once I fix this problem
    start_urls = ['http://www.alexa.com/siteinfo/google.com/',
                  'https://www.alexa.com/siteinfo/reddit.com']

    def parse(self, response):
        rank = response.css(".rankmini-rank::text").extract()
        url_raw = [response.request.url for i in range(len(rank))]
        #extract content into rows
        for item in zip(url_raw, rank):
            scraped_info = {
                str('url_raw'): item[0],
                'rank': item[1]
            }
            yield scraped_info
user_164297800
Related