I am scraping this website: https://www.eafo.eu/alternative-fuels/electricity/charging-infra-stats
I am unable to pull the dynamic chart values using bs4 or Selenium. I can get the html but no data values. When I use Selenium I am able to capture the html but without the data. Is there anything I am missing to grab this or a stronger tool that can manipulate dynamic web 
pages?
Yea, this is an interesting problem that can actually trick many people when web scraping for data... The issue is that the chart is loaded after document ready in JavaScript, you can learn more about doc ready here. But essentially, the charts are rendered after all the HTML, CSS and JS has loaded, and the data is bound to a data- attr.
I created a code example that uses NodeJS Express server to return the data in all of the charts in JSON. Essentially, it hits the URL, targets the class the charts are on, and then looks for the data-* attr that contains all the data of the chart. This way you will have working code to use and fork, if and when these situations arise with JavaScript based chart rendering.
GitHub repo with NodeJS and Python solution: https://github.com/joehoeller/dynamic-chart-parser-for-webscraping
Each of the six charts on the page are populated with data from individual API calls which can be found under the network settings of your browser. You can send requests to these endpoints yourself and parse the responses:
import urllib.parse, requests, json
headers = {'authority': 'www.eafo.eu', 'pragma': 'no-cache', 'cache-control': 'no-cache', 'sec-ch-ua': '"Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"', 'accept': 'application/json, text/javascript, */*; q=0.01', 'x-requested-with': 'XMLHttpRequest', 'sec-ch-ua-mobile': '?0', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36', 'sec-fetch-site': 'same-origin', 'sec-fetch-mode': 'cors', 'sec-fetch-dest': 'empty', 'referer': 'https://www.eafo.eu/alternative-fuels/electricity/charging-infra-stats', 'accept-language': 'en-US,en;q=0.9', 'cookie': 'yearFilter=2020; activeSubMenu=electricity; subMenuActiveItem=charging_infra_stats; fuelFilter=Electricity; _ga=GA1.2.1782486955.1628797896; _gid=GA1.2.47726291.1628797896; _gat_gtag_UA_129775638_1=1'}
params = (('compare', 'false'),)
urls = ['https://www.eafo.eu/normal-and-fast-charge-points/-1/-1/-1/false/false/nvt?compare=false', 'https://www.eafo.eu/charging-positions-per-10-evs/-1/-1/-1/false/false/nvt?compare=false', 'https://www.eafo.eu/normal-power-charging-positions/-1/-1/-1/false/false/nvt?compare=false', 'https://www.eafo.eu/fillingstations-electricity-top-5/-1/-1/-1/false/false/nvt?compare=false', 'https://www.eafo.eu/fast-charging/-1/-1/-1/false/false/nvt?compare=false', 'https://www.eafo.eu/top-5-countries-charging-positions-per-10-evs/-1/-1/-1/false/false/nvt?compare=false']
data = [[urllib.parse.urlparse(url).path.split('/')[1], json.loads(requests.get(url, headers=headers, params=params).text)] for url in urls]
result = {a:[[i['c'][0]['v'], i['c'][1]['v']] for i in b['data']['rows']] for a, b in data}
Output:
{'normal-and-fast-charge-points': [[2008, 0], [2009, 0], [2010, 0], [2011, 13], [2012, 257], [2013, 751], [2014, 1474], [2015, 3396], [2016, 5190], [2017, 8723], [2018, 11138], [2019, 15136], [2020, 24987]], 'charging-positions-per-10-evs': [['2008', 0], ['2009', 0], ['2010', '14'], ['2011', '6'], ['2012', '3'], ['2013', '4'], ['2014', '5'], ['2015', '5'], ['2016', '5'], ['2017', '5'], ['2018', '6'], ['2019', '7'], ['2020', '9']], 'normal-power-charging-positions': [['2008', 0], ['2009', 0], ['2010', 400], ['2011', 2379], ['2012', 10250], ['2013', 17093], ['2014', 24917], ['2015', 44786], ['2016', 70012], ['2017', 97287], ['2018', 107446], ['2019', 148880], ['2020', 199250]], 'fillingstations-electricity-top-5': [['Netherlands', 66461], ['France', 45413], ['Germany', 43633], ['Sweden', 13564], ['Italy', 13214]], 'fast-charging': [['2008', 0], ['2009', 0], ['2010', 0], ['2011', 13], ['2012', 257], ['2013', 751], ['2014', 1474], ['2015', 3396], ['2016', 5190], ['2017', 8723], ['2018', 11138], ['2019', 15136], ['2020', 24987]], 'top-5-countries-charging-positions-per-10-evs': [['Latvia', '3.15'], ['Slovakia', '4.34'], ['Croatia', '5.14'], ['Estonia', '5.31'], ['Netherlands', '5.71']]}
In a cleaner JSON format:
t = {' '.join(map(str.capitalize, a.split('-'))):b for a, b in result.items()}
print(json.dumps(t, indent=4))
Output:
{
"Normal And Fast Charge Points": [
[
2008,
0
],
[
2009,
0
],
[
2010,
0
],
[
2011,
13
],
[
2012,
257
],
[
2013,
751
],
[
2014,
1474
],
[
2015,
3396
],
[
2016,
5190
],
[
2017,
8723
],
[
2018,
11138
],
[
2019,
15136
],
[
2020,
24987
]
],
"Charging Positions Per 10 Evs": [
[
"2008",
0
],
[
"2009",
0
],
[
"2010",
"14"
],
[
"2011",
"6"
],
[
"2012",
"3"
],
[
"2013",
"4"
],
[
"2014",
"5"
],
[
"2015",
"5"
],
[
"2016",
"5"
],
[
"2017",
"5"
],
[
"2018",
"6"
],
[
"2019",
"7"
],
[
"2020",
"9"
]
],
"Normal Power Charging Positions": [
[
"2008",
0
],
[
"2009",
0
],
[
"2010",
400
],
[
"2011",
2379
],
[
"2012",
10250
],
[
"2013",
17093
],
[
"2014",
24917
],
[
"2015",
44786
],
[
"2016",
70012
],
[
"2017",
97287
],
[
"2018",
107446
],
[
"2019",
148880
],
[
"2020",
199250
]
],
"Fillingstations Electricity Top 5": [
[
"Netherlands",
66461
],
[
"France",
45413
],
[
"Germany",
43633
],
[
"Sweden",
13564
],
[
"Italy",
13214
]
],
"Fast Charging": [
[
"2008",
0
],
[
"2009",
0
],
[
"2010",
0
],
[
"2011",
13
],
[
"2012",
257
],
[
"2013",
751
],
[
"2014",
1474
],
[
"2015",
3396
],
[
"2016",
5190
],
[
"2017",
8723
],
[
"2018",
11138
],
[
"2019",
15136
],
[
"2020",
24987
]
],
"Top 5 Countries Charging Positions Per 10 Evs": [
[
"Latvia",
"3.15"
],
[
"Slovakia",
"4.34"
],
[
"Croatia",
"5.14"
],
[
"Estonia",
"5.31"
],
[
"Netherlands",
"5.71"
]
]
}