Crawling through Web-pages that have categories

user_10596631 0
Viewed 281

I'm trying to scrap a website that has a uncommon web-page structure, page upon page upon page until i get to the item i'm trying to extract data from,

edit(Thanks to the answers, I have been able to extract most data I require, however I need the path links to get to the said product)

Here's the code I have so far:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class MySpider(CrawlSpider):

    name = 'drapertools.com'
    start_urls = ['https://www.drapertools.com/category/0/Product%20Range']

    rules = (
        Rule(LinkExtractor(allow=['/category-?.*?/'])),
        Rule(LinkExtractor(allow=['/product/']), callback='parse_product'),
    )

    def parse_product(self, response):

        yield {
            'product_name': response.xpath('//div[@id="product-title"]//h1[@class="text-primary"]/text()').extract_first(),
            'product_number': response.xpath('//div[@id="product-title"]//h1[@style="margin-bottom: 20px; color:#000000; font-size: 23px;"]/text()').extract_first(),
            'product_price': response.xpath('//div[@id="product-title"]//p/text()').extract_first(),
            'product_desc': response.xpath('//div[@class="col-md-6 col-sm-6 col-xs-12 pull-left"]//div[@class="col-md-11 col-sm-11 col-xs-11"]//p/text()').extract_first(),
            'product_path': response.xpath('//div[@class="nav-container"]//ol[@class="breadcrumb"]//li//a/text()').extract(),
            'product_path_links': response.xpath('//div[@class="nav-container"]//ol[@class="breadcrumb"]//li//a/href()').extract(),
        }

I don't know if this would work or anything, can anyone please help me here? I would greatly appreciate it.

More Info: I'm trying to access all categories and all items within them however there is a categories within them and even more before I can get to the item.

I'm thinking of using Guillaume's LinkExtractor Code but i'm not sure that is supposed to be used for the outcome I want...

rules = (
        Rule(LinkExtractor(allow=['/category-?.*?/'])),
        Rule(LinkExtractor(allow=['/product/']), callback='parse_product'),
    )
scrapy-spider scrapy python
2 Answers

You have the same structure for all pages, maybe you can shorten it?

import scrapy

class DraperToolsSpider(scrapy.Spider):
    name = 'drapertools_spider'
    start_urls = ["https://www.drapertools.com/category/0/Product%20Range"]


    def parse(self, response):
        # this will call self.parse by default for all your categories
        for url in response.css('.category p a::attr(href)').extract():
            yield scrapy.Request(response.urljoin(url))  

        # here you can add some "if" if you want to catch details only on certain pages
        for req in self.parse_details(response):
            yield req

    def parse_details(self, response):
        yield {}
user_71068500

Why not using a CrawlSpider instead! It's perfect for this use-case!

It basically gets all the links for every page recursively, and calls a callback only for the interesting ones (I'm assuming that you are interested in the products).

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class MySpider(CrawlSpider):

    name = 'drapertools.com'
    start_urls = ['https://www.drapertools.com/category/0/Product%20Range']

    rules = (
        Rule(LinkExtractor(allow=['/category-?.*?/'])),
        Rule(LinkExtractor(allow=['/product/']), callback='parse_product'),
    )

    def parse_product(self, response):

        yield {
            'product_name': response.xpath('//div[@id="product-title"]//h1[@class="text-primary"]/text()').extract_first(),
        }
user_8697640
Related