Spider is just crawling but not scraping why?

Fustavo Gringe 0
Viewed 37

I'm trying to crawl this website

https://www.ebay.com/sch/i.html?_dmd=2&_dkr=1&iconV2Request=true&_ssn=a2z_prime_auto_parts&store_name=a2zprimeautoparts&_oac=1&_pgn=1

I'm trying to go into each product and fetch its name and price and other stuffs but I'm facing an issue that is new to me.

enter image description here

there are totally 1800+ products , and all have same xpaths which I want to scrape. but it only scraped 96. what could be the issue?

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class AutopartSpider(CrawlSpider):
    name = 'Autopart'
    allowed_domains = ['www.ebay.com']
    start_urls = ['https://www.ebay.com/sch/i.html?_dmd=2&_dkr=1&iconV2Request=true&_ssn=a2z_prime_auto_parts&store_name=a2zprimeautoparts&_oac=1']

    rules = (
        Rule(LinkExtractor(restrict_xpaths="//div[@class ='s-item__info clearfix']/a"), callback='parse_item', follow=True),
        Rule(LinkExtractor(restrict_xpaths="//a[@class='pagination__next icon-link']"))
    )

    def parse_item(self, response):
        yield{
            'part_name':response.xpath("//div[@class='vim x-item-title']/h1/span/text()").get()
        }
scrapy-splash web-crawler scrapy web-scraping python-3.x
1 Answers

Your code is working fine but you are facing a little bit problem that's why you are getting response status 200/400 which is follow = True is not in the right place.You have to place that in the pagination Rules instead.

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class AutopartSpider(CrawlSpider):
    name = 'Autopart'
    allowed_domains = ['www.ebay.com']
    start_urls = ['https://www.ebay.com/sch/i.html?_dmd=2&_dkr=1&iconV2Request=true&_ssn=a2z_prime_auto_parts&store_name=a2zprimeautoparts&_oac=1&_pgn=1']

    rules = (
        Rule(LinkExtractor(restrict_xpaths="//div[@class ='s-item__info clearfix']/a"), callback='parse_item'),
        Rule(LinkExtractor(restrict_xpaths="//a[@class='pagination__next icon-link']"),follow=True)
    )

    def parse_item(self, response):
        yield{
            'part_name':response.xpath("//div[@class='vim x-item-title']/h1/span/text()").get()
        }

Output:

'part_name': 'Rear Wheel Bearing & Hub Assembly For Cadillac Fleetwood 1988-1990 4-Wheel ABS'}
2022-09-07 20:35:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.ebay.com/itm/374136029760?hash=item571c3ec240:g:WrsAAOSwd3lirLyQ> (referer: https://www.ebay.com/sch/i.html?_dmd=2&_dkr=1&iconV2Request=true&_ssn=a2z_prime_auto_parts&store_name=a2zprimeautoparts&_oac=1&_pgn=1)
2022-09-07 20:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.ebay.com/itm/374136029711?hash=item571c3ec20f:g:sfwAAOSwFFRirLyM>
{'part_name': 'Front Wheel Bearing & Hub Assembly For Pontiac Grand Prix 2003-2008-0238'}
2022-09-07 20:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.ebay.com/itm/374136029965?hash=item571c3ec30d:g:QG0AAOSwRC1irLyR>
{'part_name': 'Front Wheel Bearing & Hub Assembly For Toyota Prius C 2012-2015'}
2022-09-07 20:35:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.ebay.com/itm/374136029434?hash=item571c3ec0fa:g:JSYAAOSwvspirLyH> (referer: https://www.ebay.com/sch/i.html?_dmd=2&_dkr=1&iconV2Request=true&_ssn=a2z_prime_auto_parts&store_name=a2zprimeautoparts&_oac=1&_pgn=1)
2022-09-07 20:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.ebay.com/itm/374136029760?hash=item571c3ec240:g:WrsAAOSwd3lirLyQ>
{'part_name': 'Front Wheel Bearing & Hub Assembly For Ford Taurus X 2008-2009 FWD'}
2022-09-07 20:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.ebay.com/itm/374136029434?hash=item571c3ec0fa:g:JSYAAOSwvspirLyH>
{'part_name': 'Rear Wheel Bearing & Hub Assembly For Dodge Viper 1992-1995'}
F.Hoque0
Related