Retrieving graph info web scraping

user_15993009 0
Viewed 67

I am new to web scraping and I need some help for my query. On this page https://ski-resort-stats.com/Hemsedal/, section SNOW HISTORY FOR HEMSEDAL I am trying to retrieving the infos on the graph (snow fall for each year). I am trying to start with one year (2013-2014).

I think I found the part associated in the html code: Screenshot from the html code

For that:

from bs4 import BeautifulSoup                                                                         
import requests

url="https://ski-resort-stats.com/Hemsedal/"                                              
html_content = requests.get(url).text
soup = BeautifulSoup(html_content, "lxml")                             
gdp_table = soup.find("g", attrs={"class": "highcharts-markers highcharts-series-0 highcharts-spline-series highcharts-color-0 highcharts-tracker"})                                                                       
gdp_table_data = gdp_table.tbody.find_all("path")`

But I am guetting this error: "AttributeError: 'NoneType' object has no attribute 'tbody'". Tried with other elements from the html codes but without success. Can someone help me with that ?

web-scraping beautifulsoup python
2 Answers

As @joni pointed out, the site runs javascript after it is initially loaded to populate the page with the graph data. The code below uses selenium to load the page, grab the data point elements on the graph for 2013-2014, and then hovers over each of those points so the info tooltip with the actual data is made visible:

from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
import time
d = webdriver.Chrome('/path/to/chromedriver')
d.get('https://ski-resort-stats.com/Hemsedal/')
results = []
for i in d.execute_script('''return document.querySelectorAll('g > path[fill="#7cb5ec"]')''')[:-1]:
    a = ActionChains(d)
    a.move_to_element(i).perform()
    time.sleep(0.3)
    results.append(d.execute_script('''
     function* get_hover_data(y_range){
         for (var i of document.querySelectorAll('text[x="8"][data-z-index="1"][y="18"]')){
             if (i.children.length === 5){
                 yield [i.children[0].textContent, i.children[3].textContent]
             }
         }

     }
     return [...get_hover_data('2013-2014')];
    '''))

_, *final_results = [i[0] for i in results if i]

Output:

[['45', '0'], ['46', '0'], ['47', '0'], ['48', '25'], ['49', '28'], ['50', '36'], ['51', '37'], ['52', '53,5'], ['1', '77,5'], ['2', '89,5'], ['3', '125,5'], ['4', '151,5'], ['5', '159,5'], ['6', '163,5'], ['7', '177,5'], ['8', '173,5'], ['9', '175'], ['10', '166'], ['11', '171'], ['12', '173,5'], ['13', '170'], ['14', '166'], ['15', '158,5']]
user_73267380

The data is embedded within the page in JavaScript form. You can use this example how to parse it:

import re
import json
import requests

url = "https://ski-resort-stats.com/Hemsedal/"
html_doc = requests.get(url).text

data = re.search(r"wpDataCharts\[.*?\] = ({.*})", html_doc).group(1)
data = re.sub(r"([a-z_]+):", r'"\1":', data)
data = re.sub(r'"http":', "http:", data)
data = json.loads(data)

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for series in data["render_data"]["options"]["series"]:
    print(series["name"], series["data"])

print()
print("week =", data["render_data"]["options"]["xAxis"]["categories"])

Prints:

2013-2014 [0, 0, 0, 25, 28, 36, 36.5, 53.5, 77.5, 89.5, 125.5, 151.5, 159.5, 163.5, 177.5, 173.5, 175, 166, 171, 173.5, 169.5, 166, 158.5]
2014-2015 [0, 0, 0, 52, 70, 67, 74.5, 78, 74, 88, 98, 102, 109.5, 113, 113, 109.5, 110.5, 98, 95, 95, 99, 108, 102]
2015-2016 [0, 0, 0, 11.5, 25.5, 34, 37, 52, 64, 76, 82, 73, 79, 105, 118, 120, 136, 141, 116, 116, 100, 97, 95]
2016-2017 [0, 0, 0, 42, 33, 22, 17, 31, 25, 40, 47, 22, 15, 17, 15, 20, 47, 50, 63, 59, 57, 51, 50]
2017-2018 [10, 10, 40, 66, 64, 64, 48, 67, 77, 85, 120, 120, 140, 155, 175, 175, 175, 175, 175, 168, 170, 180, 180]
2012-2013 [0, 0, 0, 61.5, 60, 61, 76.5, 90.5, 95, 85, 85, 85, 87.5, 100, 102.5, 102.5, 100.5, 104, 101, 100, 99, 97.5, 95]
2018-2019 [0, 0, 0, 0, 23, 33, 48, 49, 50, 75, 68, 68, 115, 115, 115, 80, 80, 85, 85, 110, 110, 110, 110]
2019-2020 [45, 45, 40, 80, 80, 80, 97, 107, 107, 107, 107, 113, 113, 113, 113, 118, 118, 127, 127, 0, 0, 0, 0]
week = [45, 46, 47, 48, 49, 50, 51, 52, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
user_100359850
Related