Scraping data when there is onclick event button

Question

I am trying to scrape data from "https://lol.inven.co.kr/dataninfo/match/teamList.php" (sorry the website is not in English)

When I don't clikc the orange squre button, the site does not have "detailTable"

When clicking the button, "detailTable" shows up, which I actually want to scrape.

With this dummy webpage

Since it is the dummy webpage, the request URL (POST) does not have any rule.

In this case, is the only way to scrape what appears when clicking onclick button ("detailTable") is Selenium? Is there anyway that I can use BeautifulSoup and Requests combined?

Any ideas are welcome. Thank you in advance!

user_6106791 · Answer

You can get all the data from that post request but first you need to get the matchcode values.

Then, make a POST request and parse an XML that you get back.

For example:

import re
import time

import requests
from bs4 import BeautifulSoup

headers = {
    "Content-Type": "application/x-www-form-urlencoded;charset=utf-8",
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0"
}

with requests.Session() as s:
    soup = (
        BeautifulSoup(
            s.get(
                "https://lol.inven.co.kr/dataninfo/match/teamList.php",
                headers=headers,
            ).text,
            "lxml"
        ).select("#lolDbMatchTeamList .listDetail a")
    )

match_ids = [re.search(r"(\d+)", a["onclick"]).group(1) for a in soup]

for match_id in match_ids[:1]:  # just getting one match here
    r = s.post(
        f"https://lol.inven.co.kr/dataninfo/match/match_detail.xml.php?dummy={int(time.time())}",
        data={"matchcode": match_id},
    )
    player_names = (
        "
".join(
            p.find("playername").getText() for p
            in BeautifulSoup(r.text, features="xml").select("player")
        )
    )
    print(player_names)

This prints player names for the first match.

RNG Ming
RNG Wei
RNG Xiaohu
RNG GALA
RNG Ming
T1 Zeus
T1 Oner
T1 Faker
T1 Gumayusi
T1 Keria

And here's a full XML that you get on each POST request.



    7345
        
            12506211412Royal Never Give Up900165RNG MingMing155Gwen_Square_0그웬lol_spell_ghost유체화lol_spell_teleport순간이동305533.669.6    
        
            12506311412Royal Never Give Up9002581RNG WeiWei34LeeSin_Square_0_2리 신lol_spell_flash점멸lol_spell_smite강타21118721113    
        
            12506411412Royal Never Give Up99012RNG XiaohuXiaohu113Lissandra_Square_0_1리산드라lol_spell_flash점멸lol_spell_teleport순간이동221080156    
        
            12506511412Royal Never Give Up9002583RNG GALAGALA64Tristana_Square_0_4트리스타나lol_spell_cleanse정화lol_spell_flash점멸5035363.69.6   
        
            12506611412Royal Never Give Up900165RNG MingMing135Rakan_Square_0라칸lol_spell_ignite점화lol_spell_flash점멸308733.69.613.2 
        
            12506722223T19002552T1 ZeusZeus100Jayce_Square_0_1제이스lol_spell_flash점멸lol_spell_teleport순간이동140330.300.3  
        
            12506822223T19002559T1 OnerOner80MonkeyKing_Square_0_2오공lol_spell_flash점멸lol_spell_smite강타1221000.511.5   
        
            12506922223T1135T1 FakerFaker89Ahri_Square_0_2아리lol_spell_cleanse정화lol_spell_flash점멸0526700.40.4  
        
            12507022223T19002255T1 GumayusiGumayusi129Jhin_Square_0진lol_spell_exhaust탈진lol_spell_flash점멸11167112  
        
            12507122223T1900280T1 KeriaKeria144yummi_square_0유미lol_spell_exhaust탈진lol_spell_heal회복0326700.70.7

Scraping data when there is onclick event button

1 Answers