I'm using Scrapy to scrape data from nonfungible.com:
import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class Nft2Spider(CrawlSpider):
name = 'nft2'
allowed_domains = ['nonfungible.com']
start_urls = ['https://nonfungible.com/market/history/cryptopunks?filter=nftTicker%3D%25CF%25BE&filter=saleType%3D&length=15000&sort=blockTimestamp%3Ddesc&start=0']
rules = (Rule (LinkExtractor(allow='/project/cryptopunks/*'), callback="parse_obj", follow=True),
)
def parse_obj(self, response):
img = response.xpath('//*[@id="__next"]/div/div/section/div[1]/div/div[2]/div[1]/div/div/div[2]/span/div/div/img/@src').extract()
num = response.xpath('//*[@id="__next"]/div/div/section/div[1]/div/div[2]/div[2]/div/div/h1/text()').extract()
transact = response.xpath('//*[@class="styles__TableRow-sc-1buchb9-10 juIjAX"]')
for tran in transact:
priceUSD = tran.xpath('div[2]/div/div/text()').extract()
priceETH = tran.xpath('div[3]/div/div/text()').extract()
saleDate = tran.xpath('div[4]/div/div/text()').extract()
print(priceUSD)
print(priceETH)
print(saleDate)
I am having two problems. First, the crawler is only visiting a handful of links - the first 10, it looks like. Second, all the values (priceUSD, priceETH, saleDate) return [].
Any help is greatly appreciated.