I am trying to scrape lol.fandom (cblol, Brazilian, stats or any stats really tbh) for a college project but all I get from this website is "NaN". I have no idea how to get around it.
I failed to locate any API, also. Can someone help me to scrape this? If I have to use another language, no problem, I can learn it. If I shouldn't be scraping this website, how can I know in the future a sign that "the website doesn't want to be scraped"?
Code from "Match_History":
import requests
from bs4 import BeautifulSoup
import pandas as pd
cblol_url = "https://lol.fandom.com/wiki/CBLOL/2022_Season/Split_2/Match_History"
data = requests.get(cblol_url)
soup = BeautifulSoup(data.text)
cblol_table = soup.select('table.wikitable')
matches_cblol = pd.read_html(data.text, match="Tournament") [0]
matches_cblol
Result - a bunch of NaNs:
Tournament: CBLOL/2022 Season/Split 2; Limit: 200 - Open As Query Date P Blue Red Winner Bans Bans.1 Picks Picks.1 Blue Roster Red Roster SB VOD 0 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 1 2022-08-07 12.14 NaN NaN NaN NaN Robo, Croc, tinowns, Brance, Ceos fNb, Goot, Envy, Netuno, RedBert SB VOD 2 2022-08-07 12.14 NaN NaN NaN NaN KiaRi, Disamis, Krastyel, Matsukaze, Cavalo Zecas, Erasus, evr0t, NinjaKiwi, Mido SB VOD 3 2022-08-07 12.14 NaN NaN NaN NaN Hidan, Yampi, NOsFerus, micaO, Jockster GUIGO, Aegis, Grevthar, TitaN, JoJo SB VOD 4 2022-08-07 12.14 NaN NaN NaN NaN Tay, Ranger, Tutsz, Flare, Wos Wizer, CarioK, dyNquedo, Trigo, Damage SB VOD ... ... ... ... ... ... ... ... ... ... ... ... ... ... 86 2022-06-11 12.10 NaN NaN NaN NaN Tay, Geum go, Tutsz, Flare, Kuri Robo, Croc, tinowns, Brance, Ceos SB VOD 87 2022-06-11 12.10 NaN NaN NaN NaN fNb, Goot, Envy, Netuno, RedBert Parang, Wiz, hauz, DudsTheBoy, Scuro SB VOD 88 2022-06-11 12.10 NaN NaN NaN NaN KiaRi, Disamis, Krastyel, Matsukaze, Cavalo DoRun, Hugato, Anyyy, Celo, Sive SB VOD 89 2022-06-11 12.10 NaN NaN NaN NaN Hidan, Yampi, NOsFerus, micaO, Jockster Trap, Minerva, Goku, NinjaKiwi, Mocha SB VOD 90 2022-06-11 12.10 NaN NaN NaN NaN Wizer, CarioK, dyNquedo, Trigo, Damage GUIGO, Aegis, Grevthar, TitaN, JoJo SB VOD 91 rows × 13 columns
I also tried scraping other links to see if it was just a "Match_History" problem but when I tried to scrape "wiki/CBLOL/2022_Season/Split_2", for example, a more general view on the tournament than "wiki/CBLOL/2022_Season/Split_2/Match_History:
from bs4 import BeautifulSoup
import requests
cblol_url = "https://lol.fandom.com/wiki/CBLOL/2022_Season/Split_2"
data = requests.get(cblol_url)
soup = BeautifulSoup(data.text)
cblol_table2 = soup.select('table.wikitable')
cblol_table2[0:8]
This was the furthest I could get from this URL. I can't get pandas to show me a table after this steps if my life depended on it.
Please help.