Trouble with webscraping (Null list)

user_15477491 0
Viewed 43

I am currently trying to get the ranks of a players but always come back with a null list. Been struggling for a while, would really appreciate some help and any tips for future project to tackle these issues. Or places to better understand beautifulsoup in general

import requests as req
from bs4 import BeautifulSoup

id = "epic"
tag = "random"
url = f"https://rocketleague.tracker.network/rocket-league/profile/{id}/{tag.lower()}/overview"
html = req.get(url).content
soup = BeautifulSoup(html,"lxml")
line = soup.findAll("div",{"class":"rank"})
print(line)

Here's what I'm trying to get:

Screenshot showing desired element

web-scraping python
1 Answers

Loading response with the requests doesn't work here, simply because the site loads contents dynamically & thus you need to use some webdriver.

Here's what I've come up with:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(executable_path=r"driver/geckodriver.exe", options=options)

id = "epic"
tag = "random"
url = f"https://rocketleague.tracker.network/rocket-league/profile/{id}/{tag.lower()}/overview"

driver.get(url)
driver.implicitly_wait(2) #allow some time to fully load, you may tweak accordingly
ranks = driver.find_elements_by_css_selector(r'[class="rank"]')

for i in ranks:
    print(i.text)
driver.quit()

Resulting in:

#695,273 • Top 32%
#1,409,786 • Bottom 26%
#1,240,839 • Bottom 43%
#1,195,373 • Bottom 45%
#875,794 • Top 41%
#874,338 • Top 41%
#1,530,195 • Bottom 29%
#960,411 • Top 45%
Unranked Division I
#1,663,447 • Top 33%
Gold III Division III
#2,974,386 • Bottom 29%
Platinum II Division III
#2,879,016 • Bottom 40%
Platinum II Division II
#2,363,741 • Top 50%
Gold III Division III
#2,407,466 • Bottom 42%
Platinum II Division I
#2,054,002 • Top 48%
Gold II Division II
#2,321,030 • Bottom 41%
user_72671500
Related