Crawling through multiple links on Scrapy

user_11319452 0
Viewed 568

I'm trying to first crawl through the main page of this website for the links to a table for each year. Then I'd like to scrape each site, while maintaining record of each year.

So far I have my spider constructed as:

div = response.xpath('//*[@id="sidebar"]/div[1]/nav/ul/li[5]/div')
    
hrefs = div.xpath('*//a').extract()
splits = {}
    
for href in hrefs:
    split = href.split('"')
    link = split[1]
    date = split[2]
    clean_date = "".join(re.findall("[^><a/]",date))
    clean_link = "http://www.ylioppilastutkinto.fi" + str(link)
    splits[clean_date] = clean_link

I would then like to go through each link in this file and crawl through them, using the following logic:

table = resp.xpath('//*[@id="content"]/table/tbody')
rows = table.xpath('//tr')
        
data_dict = {"Category": 
            [w3lib.html.remove_tags(num.get()) for num in rows[0].xpath('td')[1:]]
            }

for row in rows[1:]:
    data = row.xpath('td')
    title = w3lib.html.remove_tags(data[0].get())
    nums = [w3lib.html.remove_tags(num.get()) for num in data[1:]]
    data_dict[title] = nums

My problem is that I couldn't find a way to do this effectively. Calling scrapy.Request on the url returns a response with just the content <html></html>. If there was a way where the response object could resemble the one given by the fetch command in Scrapy shell that would be ideal, since I've based the selection logic on testing with that command.

Edit:

Here's the entire spider so far

The idea is the run the first for loop to get the link and then the second for loop to extract the tables from said links.

import scrapy
import regex as re
from scrapy.http import HtmlResponse
import w3lib.html

class MainSpider(scrapy.Spider):
    name = 'links'
    allowed_domains = ['www.ylioppilastutkinto.fi/ylioppilastutkinto/pisterajat']
    start_urls = ['https://www.ylioppilastutkinto.fi/ylioppilastutkinto/pisterajat/']

    def parse(self, response):
        div = response.xpath('//*[@id="sidebar"]/div[1]/nav/ul/li[5]/div')
        
        hrefs = div.xpath('*//a').extract()
        splits = {}
        
        for href in hrefs:
            split = href.split('"')
            link = split[1]
            date = split[2]
            clean_date = "".join(re.findall("[^><a/]",date))
            clean_link = "http://www.ylioppilastutkinto.fi" + str(link)
            splits[clean_date] = clean_link

        
        for date,url in splits.items():
            resp = HtmlResponse(url)
            
            table = resp.xpath('//*[@id="content"]/table/tbody')
            rows = table.xpath('//tr')
        
            data_dict = {"Category":[w3lib.html.remove_tags(num.get()) for num in rows[0].xpath('td')[1:]]}

            for row in rows[1:]:
                data = row.xpath('td')
                title = w3lib.html.remove_tags(data[0].get())
                nums = [w3lib.html.remove_tags(num.get()) for num in data[1:]]
                data_dict[title] = nums
                
        
                yield {
                    'Date': date,
                    'Scores': data_dict}
scrapy web-scraping xpath html python
1 Answers

Initializing a HtmlResponse(url) doesn't accomplish anything, since the class doesn't make the request itself.

To add a request to scrapy's scheduler, you need to yield one, eg: yield scrapy.Request(url, callback=self.parse).

That being said, there are many improvements you can make to your spider.

  • Use scrapy's builtin LinkExtractor instead of string splitting

  • use css selectors instead of the hardcoded xpaths

  • use selector.root.text instead of w3lib.remove_tags (to remove the dependency entirely)

Here is a working example:

import scrapy
from scrapy.linkextractors import LinkExtractor


class MainSpider(scrapy.Spider):
    name = 'links'
    allowed_domains = ['www.ylioppilastutkinto.fi']
    start_urls = ['https://www.ylioppilastutkinto.fi/ylioppilastutkinto/pisterajat/']

    def parse(self, response):
        le = LinkExtractor(
            allow_domains=self.allowed_domains,
            restrict_xpaths='//*[@id="sidebar"]/div[1]/nav/ul/li[5]/div',
        )
        for link in le.extract_links(response):
            yield scrapy.Request(
                url=link.url,
                callback=self.parse_table,
                cb_kwargs={ 'date': link.text },
            )

    def parse_table(self, response, date):
        rows = response.css('#content table tbody tr')
        if not rows:
            print(f'No table found for url: {response.url}')
            return

        category = [char.root.text for char in rows[0].css('td strong')[1:]]
        if not category:
            category = [char.root.text for char in rows[0].css('td')[1:]]

        for row in rows[1:]:
            cols = row.css('td')
            title = cols[0].root.text
            nums = [col.root.text for col in cols[1:]]
            yield {
                'Date': date,
                'Category': category,
                title: nums
            }

Note that your category parsing doesn't appear to work. I'm not exactly sure what you are trying to extract, so I'll leave that one for you.

user_107462240
Related