how to return javascript result with scrapy-spash

Viewed 18

I am trying to return the result after executing a javascript. The javascript returns the data of an interactive chart from the url https://www.transfermarkt.com/erling-haaland/marktwertverlauf/spieler/418560.

With Selenium, my code is:

driver.get(response.url)
driver.implicitly_wait(5)
temp = driver.execute_script('return window.Highcharts.charts[0]''.series[0].options.data')
driver.close()
data = [item for item in temp] 

And it returns a list of all the data:

[{'age': '16',
  'datum_mw': 'Dec 18, 2016',
  'marker': {'symbol': 'url(https://tmssl.akamaized.net/images/wappen/verysmall/1057.png?lm=1480871779)'},
  'mw': '€200Th.',
  'verein': 'Bryne FK',
  'x': 1482015600000,
  'y': 200000},
 {'age': '17',
  'datum_mw': 'Dec 23, 2017',
  'marker': {'symbol': 'url(https://tmssl.akamaized.net/images/wappen/verysmall/687.png?lm=1409159512)'},
  'mw': '€300Th.',
  'verein': 'Molde FK',
  'x': 1513983600000,
  'y': 300000}
...

I want to replicate this with scrapy-splash as it is much faster and more efficient that selenium. What I tried:

javascript_script = """
return window.Highcharts.charts[0]''.series[0].options.data;
"""

class TestSpider(scrapy.Spider):
    name = 'test_spider'
    

    def start_requests(self):
        url = 'https://www.transfermarkt.com/erling-haaland/marktwertverlauf/spieler/418560'
        yield SplashRequest(url, callback=self.parse, endpoint='render.html', headers = headers_,  
            args={
                'wait': 2,
                'js_source': javascript_script, 
                'html': 1, 
                'png': 1, 
                'width': 1000,
            })

    def parse(self, response): 
        #Here I want to return the same output as the selenium output
0 Answers
Related