Scrapy Not Following All Links & xPath Not Returning Proper Data

Viewed 29

I'm using Scrapy to scrape data from nonfungible.com:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor


class Nft2Spider(CrawlSpider):
    name = 'nft2'
    allowed_domains = ['nonfungible.com']
    start_urls = ['https://nonfungible.com/market/history/cryptopunks?filter=nftTicker%3D%25CF%25BE&filter=saleType%3D&length=15000&sort=blockTimestamp%3Ddesc&start=0']
    rules = (Rule (LinkExtractor(allow='/project/cryptopunks/*'), callback="parse_obj", follow=True),
    )
    def parse_obj(self, response):
        img = response.xpath('//*[@id="__next"]/div/div/section/div[1]/div/div[2]/div[1]/div/div/div[2]/span/div/div/img/@src').extract()
        num = response.xpath('//*[@id="__next"]/div/div/section/div[1]/div/div[2]/div[2]/div/div/h1/text()').extract()
        transact = response.xpath('//*[@class="styles__TableRow-sc-1buchb9-10 juIjAX"]')
        for tran in transact:
            priceUSD = tran.xpath('div[2]/div/div/text()').extract()
            priceETH = tran.xpath('div[3]/div/div/text()').extract()
            saleDate = tran.xpath('div[4]/div/div/text()').extract()
            print(priceUSD)
            print(priceETH)
            print(saleDate)

I am having two problems. First, the crawler is only visiting a handful of links - the first 10, it looks like. Second, all the values (priceUSD, priceETH, saleDate) return [].

Any help is greatly appreciated.

0 Answers
Related