How to correclty loop links with Scrapy?

Question

I'm using Scrapy and I'm having some problems while loop through a link.

I'm scraping the majority of information from one single page except one which points to another page.

There are 10 articles on each page. For each article I have to get the abstract which is on a second page. The correspondence between articles and abstracts is 1:1.

Here the divsection I'm using to scrape the data:


    
        
            
Article
        Climate change and agrarian struggles: an invitation to
                    contribute to a JPS Forum
        
            
                Saturnino M. Borras Jr., Ian Scoones, Amita Baviskar, Marc Edelman, Nancy Lee Peluso & Wendy Wolford
            
            Pages: 1-28
            Published online: 06
                        Aug 2021
        
        
        Abstract | Full Text | References | PDF (2239 KB) | EPUB | Permissions\xa0
        
            
                6049 Views
                0 CrossRef citations
                0Altmetric
                
            
        
Open Access

To do so I have defined the following script

from cgitb import text
import scrapy
import pandas as pd


class QuotesSpider(scrapy.Spider):
    name = "jps"

    start_urls = ['https://www.tandfonline.com/toc/fjps20/current']
    

    def parse(self, response):
        self.logger.info('hello this is my first spider')
        Title = response.xpath("//span[@class='hlFld-Title']").extract()
        Authors = response.xpath("//span[@class='articleEntryAuthorsLinks']").extract()
        License = response.xpath("//span[@class='part-tooltip']").extract()
        abstract_url = response.xpath('//*[@class="tocDeliverFormatsLinks"]/a/@href').extract()
        row_data = zip(Title, Authors, License, abstract_url)
        
        for quote in row_data:
            scraped_info = {
                # key:value
                'Title': quote[0],
                'Authors': quote[1],
                'License': quote[2],
                'Abstract': quote[3]
            }
            # yield/give the scraped info to scrapy
            yield scraped_info
    
    
    def parse_links(self, response):
        
        for links in response.xpath('//*[@class="tocDeliverFormatsLinks"]/a/@href').extract():
            yield scrapy.Request(links, callback=self.parse_abstract_page)
        #yield response.follow(abstract_url, callback=self.parse_abstract_page)
    
    def parse_abstract_page(self, response):
        Abstract = response.xpath("//div[@class='hlFld-Abstract']").extract_first()
        row_data = zip(Abstract)
        for quote in row_data:
            scraped_info_abstract = {
                # key:value
                'Abstract': quote[0]
            }
            # yield/give the scraped info to scrapy
            yield scraped_info_abstract

Authors, title and license are correctly scraped. For the Abstract I'm having the following error:

ValueError: Missing scheme in request url: /doi/abs/10.1080/03066150.2021.1956473

To check if the path was correct I removed the abstract_url from the loop:

 abstract_url = response.xpath('// [@class="tocDeliverFormatsLinks"]/a/@href').extract_first()
 self.logger.info('get abstract page url')
 yield response.follow(abstract_url, callback=self.parse_abstract)

I can correctly reach the abstract corresponding to the first article, but not the others. I think the error is in the loop.

How can I solve this issue?

Thanks

user_1012157 · Answer

As @tgrnie has explained, the URL is a relative URL, which needs to be converted to an absolute URL.

Scrapy has a wrapper around urljoin, which is simply response.urljoin(). No additional imports are required. See official docs here.

So this line:

yield scrapy.Request(links, callback=self.parse_abstract_page)

can be modified like this:

yield scrapy.Request(response.urljoin(links), callback=self.parse_abstract_page)

The other approach is to use response.follow as you used in your code:

 yield response.follow(abstract_url, callback=self.parse_abstract)

If you want to follow all linked, use yield from along with follow_all as the following example:

yield from response.follow_all(list_of_urls, callback=self.parse_abstract)

The biggest difference between yield Request(URL) and yield response.follow(url) is that a relative URL will work with response.follow, while you must provide a complete URL to create a Request object.

See the documentation here.

user_12734467 · Answer

The link to the article abstract appears to be a relative link (from the exception). /doi/abs/10.1080/03066150.2021.1956473 doesn't start with https:// or http://.

You should append this relative URL to the base URL of the website (i.e. if the base URL is "https://www.tandfonline.com", you can

import urllib.parse

link = urllib.parse.urljoin("https://www.tandfonline.com", link)

Then you'll have a proper URL to the resource.

How to correclty loop links with Scrapy?

2 Answers