scrape data from interactive chart using selenium, bs4 or requests

Question

I want to scrape data from charts on this page: http://188.166.44.172/match/live-stats/100941310

I tried requests and bs4, but failed to get any data, I also tried with selenium and no data as well.

Here's the code using requests:

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0'}

session = requests.Session()
r = session.get(u, timeout=30, headers=headers)    

soup = BeautifulSoup(r.content, 'html.parser')

for i in soup.find_all('rect'):
    if i.has_attr("onmouseover"):
        text = i.get('onmouseover')
        print(text)

And the code using selenium:

from selenium import webdriver
from bs4 import BeautifulSoup
u = "http://188.166.44.172/match/live-stats/100941310"

driver = webdriver.Chrome(executable_path=r"C:/chromedriver.exe", options=options)

driver.get(url)

soup = BeautifulSoup(driver.page_source, 'html.parser')

for i in soup.find_all('rect'):   #I also tried soup.select('*')
    if i.has_attr("onmouseover"):
        text = i.get('onmouseover')
        print(text)

Is there's any way to scrape data from those charts using python ?

user_6106791 · Answer

The reason you're not getting anything back is because all the charts are generated dynamically by JavaScript and with, for example, bs4 you won't read any of this.

However, the data for the charts is embedded in the HTML. You could parse that and plot.

Here's how:

import ast
import re

import requests
import matplotlib.pyplot as plt

target_url = "http://188.166.44.172/match/live-stats/100941310"
page_source = requests.get(target_url).text

raw_attack_data = ast.literal_eval(
    re.search(r"var all_attack = ($$.*$$)", page_source).group(1),
)
all_attack = [i[1] for i in raw_attack_data if isinstance(i, list)]

plt.plot(all_attack, label="attack")
plt.legend(loc="lower right")
plt.show()

This should give you a plot like this:

As I've said, eveyrthing you need is in the source code, so you'd have to play around with the values.

The source looks like this:

Where the first value of the inner list is the game time and the second value is the stat that's plotted on the charts.

Note that some arrays have values in {}. These' are those icon markers on the charts. You can filter them out with isistance(i, list) since these can be easily parsed as dicts, as I've shown above.

EDIT:

Yes, it's possible to get division and team info, as everything is in the HTML. I've reworked the initial answer a bit and came up with this:

import ast
import re

import requests
from bs4 import BeautifulSoup
import matplotlib.pyplot as plt

target_url = "http://188.166.44.172/match/live-stats/100941310"
page_source = requests.get(target_url).text


def get_match_info(html_source: str) -> list:
    return [
        i.getText(strip=True) for i
        in BeautifulSoup(html_source, "lxml").select("h1 a")
    ]


def get_stats(html_source: str, search_str: str) -> tuple:
    raw_data = ast.literal_eval(
        re.search(fr"var {search_str} = ($$.*$$)", html_source).group(1),
    )
    filtered = [i[1] for i in raw_data if isinstance(i, list)]
    game_time = [i[0] for i in raw_data if isinstance(i, list)]
    return game_time, filtered


division, home, away = get_match_info(page_source)

time_, attack_home = get_stats(page_source, "dangerous_home")
_, attack_away = get_stats(page_source, "dangerous_away")

plt.suptitle(f"{division} - {home} v {away}")
plt.ylabel("Attack")
plt.xlabel("Game time")
plt.plot(time_, attack_home, color="blue", label=home)
plt.plot(time_, attack_away, color="black", label=away)
plt.legend(loc="lower right")
plt.show()

This produces a plot:

scrape data from interactive chart using selenium, bs4 or requests

1 Answers