BeautifulSoup WebScraping Issue: Cannot find specific classes for this specific Website (Python 3.7)

user_15697197 0
Viewed 161

I am a bit new to webscraping, I have created webscrapers with the methods below before, however with this specific website I am running into an issue where the parser cannot locate the specific class ('mainTitle___mbpq1') this is the class which refers to the text of announcement. Whenever I run the code it returns None. This also the case for the majority of other classes. I want to capture this info without using selenium, since this slows the process down from what I understand. I think the issue is that it is a json file, and so script tags are being used (I may be completely wrong, just a guess), but I do not know much about this area, so any help would be much appreciated.

The code below I have attempted using, with no success.

from bs4 import BeautifulSoup
import re
import requests
# Method 1
url_4 = "https://www.kucoin.com/news/categories/listing"
res = requests.get(url_4)
html_page = res.content
soup = BeautifulSoup(html_page, 'html.parser')
texts = soup.body
text = soup.body.div.find('div',{'class':'mainTitle___mbpq1'})
print(text)


from bs4 import BeautifulSoup
import urllib3
import re
# Method2
http = urllib3.PoolManager()
comm = re.compile("<!--|-->")
def make_soup(url):
    page = http.request('GET', url)
    soupdata = BeautifulSoup(page.data,features="lxml")
    return soupdata

soup = make_soup(url_4)
Annouce_Info = soup.find('div',{'class':'mainTitle___mbpq1'})

print(Annouce_Info)

linkKuCoin Listing

web-scraping beautifulsoup python-3.x
2 Answers

The data is loaded from external source via Javascript. To print all article titles, you can use this example:

import json
import requests


url = "https://www.kucoin.com/_api/cms/articles"
params = {"page": 1, "pageSize": 10, "category": "listing", "lang": ""}
data = requests.get(url, json=params).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for item in data["items"]:
    print(item["title"])

Prints:

PhoenixDAO (PHNX) Gets Listed on KuCoin!
LABS Group (LABS) Gets Listed on KuCoin! World Premiere!
Polkadex (PDEX) Gets Listed on KuCoin! World Premiere!
Announcement of Polkadex (PDEX) Token Sale on KuCoin Spotlight
KuCoin Futures Has Launched USDT Margined NEO, ONT, XMR, SNX Contracts
Introducing the Polkadex (PDEX) Token Sale on KuCoin Spotlight
Huobi Token (HT) Gets Listed on KuCoin!
KuCoin Futures Has Launched USDT Margined XEM, BAT, XTZ, QTUM Contracts
RedFOX Labs (RFOX) Gets Listed on KuCoin!
Boson Protocol (BOSON) Gets Listed on KuCoin! World Premiere!
user_100359850

If you trying to scrape information about new listings at crypto exchanges, you can be interested in this API:

https://rapidapi.com/Diver44/api/new-cryptocurrencies-listings/

import requests

url = "https://new-cryptocurrencies-listings.p.rapidapi.com/new_listings"

headers = {
    'x-rapidapi-host': "new-cryptocurrencies-listings.p.rapidapi.com",
    'x-rapidapi-key': "your-key"
    }

response = requests.request("GET", url, headers=headers)

print(response.text)

It includes an endpoint with New Listings from the biggest exchanges and a very useful endpoint with information about exchanges where you can buy specific coins and prices for this coin at that exchanges

user_173043680
Related