How to scrape a value that's computed in real time

user_7082841 0
Viewed 605

I'm trying to scrape a value from an online calculator, here the value I want, which is 5.1, has already been computed: CVSS 3 Calculator. The thing is when I write the following

page = requests.get('https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:A/AC:H/PR:L/UI:R/S:C/C:L/I:L/A:L')
print(page.content)

I see the following snippet:

<dd id="cvss-overall-score-cell">{{vm.overallScore}}</dd>\r\n

I believe this means there some computation that still needs to take place. But if I open up chrome's dev tools I can find the following:

enter image description here

Some of that text doesn't seem to translate, but I'm pretty new to this stuff so I'm not sure exactly if I using requests correctly (I'm using BeautifulSoup too, but didn't see it's use here). What I'm guessing is that the page needs a second to load in the string that triggers calculation; as in right now I think I'm scraping nonpopulated data, the page needs a second to load. So could I pause requests or something similar, or is there a better way to go?

python-requests web-scraping python
2 Answers

What you are seeing is a template expression that will be later replaced by Javascript. In looks like AngularJS template code (though the syntax is pretty common among various frameworks).

Since you are not executing Javascript, but only downloading HTML, the value is not shown.

The actual value is probably obtained through other means, like an HTTPRequest. Open your browser development tools and look at the network tab. Also review the source code. The value you look for should be along the Javascript in the page, or obtained dynamically from the server via a separate HTTP request.

user_4016560

You can use selenium and a CSS selector to grab the same value from the chart.

Each value on the right can be grabbed from the charts on the left. Example here showing Overall CVSS Score:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options  

chrome_options = Options()  
chrome_options.add_argument("--headless")  
url ="https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:A/AC:H/PR:L/UI:R/S:C/C:L/I:L/A:L"

d = webdriver.Chrome(chrome_options=chrome_options)
d.get(url)
print(d.find_element_by_css_selector("#cvss-overall-score-chart > div.jqplot-point-label.jqplot-series-0.jqplot-point-0").text)
d.quit()
user_62412350
Related