I am trying to return the result after executing a javascript. The javascript returns the data of an interactive chart from the url https://www.transfermarkt.com/erling-haaland/marktwertverlauf/spieler/418560.
With Selenium, my code is:
driver.get(response.url)
driver.implicitly_wait(5)
temp = driver.execute_script('return window.Highcharts.charts[0]''.series[0].options.data')
driver.close()
data = [item for item in temp]
And it returns a list of all the data:
[{'age': '16',
'datum_mw': 'Dec 18, 2016',
'marker': {'symbol': 'url(https://tmssl.akamaized.net/images/wappen/verysmall/1057.png?lm=1480871779)'},
'mw': '€200Th.',
'verein': 'Bryne FK',
'x': 1482015600000,
'y': 200000},
{'age': '17',
'datum_mw': 'Dec 23, 2017',
'marker': {'symbol': 'url(https://tmssl.akamaized.net/images/wappen/verysmall/687.png?lm=1409159512)'},
'mw': '€300Th.',
'verein': 'Molde FK',
'x': 1513983600000,
'y': 300000}
...
I want to replicate this with scrapy-splash as it is much faster and more efficient that selenium. What I tried:
javascript_script = """
return window.Highcharts.charts[0]''.series[0].options.data;
"""
class TestSpider(scrapy.Spider):
name = 'test_spider'
def start_requests(self):
url = 'https://www.transfermarkt.com/erling-haaland/marktwertverlauf/spieler/418560'
yield SplashRequest(url, callback=self.parse, endpoint='render.html', headers = headers_,
args={
'wait': 2,
'js_source': javascript_script,
'html': 1,
'png': 1,
'width': 1000,
})
def parse(self, response):
#Here I want to return the same output as the selenium output