how to get data data from datalayer.push using python webscraping

user_11433004 0
Viewed 358

my code us:

        # init scrapy selector
        response = Selector(text=content)
        
        json_data = json.loads(script.get() for script in (re.findall(r'dataLayer\.push\(([^)]+)'),response.css('script::text'))).group(1)
        print(json_data)
    
    # debug data extraction logic
    HummartScraper.parse_product(HummartScraper, '')

' the output error is:

Traceback (most recent call last):
  File "hummart2.py", line 86, in parse_product
    json_data = json.loads(script.get() for script in (re.findall(r'dataLayer\.push\(([^)]+)'),response.css('script::text'))).group(1)
TypeError: findall() missing 1 required positional argument: 'string'

why is this error getting.

scrapy web-scraping python
2 Answers

For a single dataLayer:

data_layer = response.css('script::text').re_first(r'dataLayer\.push\(([^)]+)')
data = json.loads(data_layer)

You can use response.css(...).re() to get a list of matches.

user_3477670

but this give me this type of error:

 File "hummart2.py", line 88, in parse_product
    data = json.loads(data_layer_raw)[1]
  File "/home/danish-khan/miniconda3/lib/python3.7/json/__init__.py", line 341, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not NoneType
user_114330040
Related