Scrapy Returns 'None' after the 10th row in table

user_15416774 0
Viewed 123

I've tried many different xpath expressions but can't quite get this to work. Basically, I will get the text for the first 10 rows in a table, then 'None' for the following 90. If I do a different xpath expression (one that was suggested to me but I personally don't fully understand), it gives me the last 90 but not the first 10. At first I thought it was due to tbody, so I removed that from my xpath expression. The class for the tag I am targeting also changes after the 10th row so I'm not sure if that makes a difference. I've tried inserting an "or" statement to try to get the best of both xpaths but it seems it has been improperly placed. Please help!

class CryptocurrenciesSpider(scrapy.Spider):
    name = 'cryptocurrencies'
    allowed_domains = ['www.coinmarketcap.com']

    def start_requests(self):
        yield scrapy.Request(url='https://www.coinmarketcap.com/', callback=self.parse)


#Udemy Answer. Yields last 90 with 'None' for the first 10 rows. 
     def parse(self, response):
         for row in response.xpath("//table//tr"):
             currency = row.xpath(".//td/a/span[2]/text()").get()
             if currency:
                 yield {
                     'currency': currency
                 }

#Yields the first 10 and then 'None' for the rest.
    def parse(self, response):
        for row in response.xpath(".//table//tr"):
                yield {
                'Currency': row.xpath(".//td[3]/div/a/div/div/p/text()").get()
            }

# Tried the "or" operator to no avail.
# row.xpath(".//td[3]/div/a/div/div/p/text() or .//td/a/span[2]/text()").get(
# row.xpath(".//td[3]/div/a/div/div/p/text()" or ".//td/a/span[2]/text()").get()
scrapy web-scraping xpath python
2 Answers

Jacob,

as a matter of fact, the solution is at your hands already.

You have the xpath expressions for the first ten rows, as well as for the subsequent ninety. All you need to do is to combine them.

I suggest that you try the pipe |, which serves as a union operator.

The whole xpath to fetch the full 100 names with .getall() could be the following lenghthy beast

//table[contains(@class, "cmc-table")]//tr/td[3]//a/div/div/p/text() | //table[contains(@class, "cmc-table")]//tr/td[3]//a/span[2]/text()

Alternatively, you might want to iterate over the row/column/anchor part and use the union only on tha part that changes.

Good luck and have fun!

user_4137680

Use requests and json library. you will not have to worry about the xpath of the elements. You will get this data hassle free. something like below -

req = requests.get('https://api.coinmarketcap.com/data-api/v3/cryptocurrency/listing?start=1&limit=100&sortBy=market_cap&sortType=desc&convert=USD,btc,eth&cryptoType=all&tagType=all&aux=ath,atl,high24h,low24h,num_market_pairs,cmc_rank,date_added,tags,platform,max_supply,circulating_supply,total_supply,volume_7d,volume_30d',
            headers={'Accept': 'application/json','Content-Type': 'application/json'})
if(req.status_code == 200):
    josn2 = req.json()
    currencyDetails = josn2['data']['cryptoCurrencyList']
    print(len(currencyDetails))
    for i in currencyDetails:
        print("name : " + i.get('name'), "\t\tSymbol : "+ i.get('symbol'))

You will need following imports -

import requests
import json

Output -

100
name : Bitcoin      Symbol : BTC
name : Ethereum         Symbol : ETH
name : Binance Coin         Symbol : BNB
name : XRP      Symbol : XRP
name : Tether       Symbol : USDT
name : Cardano      Symbol : ADA
name : Polkadot         Symbol : DOT
name : Uniswap      Symbol : UNI
user_54003620
Related