Parse Script Tag after BeautySoup in Python

user_10010833 0
Viewed 1573

Hello all members of the forum. When parsing a page, I encountered the problem of extracting data from the tag script. The internal content of the tag is not an json object. Using web.driver results did not. Who faced such things? I ask for your help.

Code example:

<script>window.ShopifyAnalytics = window.ShopifyAnalytics || {};
window.ShopifyAnalytics.meta = window.ShopifyAnalytics.meta || {};
window.ShopifyAnalytics.meta.currency = 'AUD';
var meta = {"product":{"id":8993669708,"vendor":"Womanizer","type":"Vibrators","variants":[{"id":31066737740,"price":14999,"name":"Womanizer - Black","public_title":"Black","sku":"172145678"},{"id":31066737804,"price":14999,"name":"Womanizer - Purple","public_title":"Purple","sku":"172146924"},{"id":31066737868,"price":14999,"name":"Womanizer - Pink","public_title":"Pink","sku":"172150324"},{"id":31066737996,"price":14999,"name":"Womanizer - Tattoo","public_title":"Tattoo","sku":"172205168"},{"id":1509908217881,"price":14999,"name":"Womanizer - Blue","public_title":"Blue","sku":"1725205076"}]},"page":{"pageType":"product","resourceType":"product","resourceId":8993669708}};
for (var attr in meta) {
  window.ShopifyAnalytics.meta[attr] = meta[attr];
}</script>

parsing python javascript
1 Answers

Using Regex.

Demo:

from bs4 import BeautifulSoup
import json
import re


s = """<script>window.ShopifyAnalytics = window.ShopifyAnalytics || {};
window.ShopifyAnalytics.meta = window.ShopifyAnalytics.meta || {};
window.ShopifyAnalytics.meta.currency = 'AUD';
var meta = {"product":{"id":8993669708,"vendor":"Womanizer","type":"Vibrators","variants":[{"id":31066737740,"price":14999,"name":"Womanizer - Black","public_title":"Black","sku":"172145678"},{"id":31066737804,"price":14999,"name":"Womanizer - Purple","public_title":"Purple","sku":"172146924"},{"id":31066737868,"price":14999,"name":"Womanizer - Pink","public_title":"Pink","sku":"172150324"},{"id":31066737996,"price":14999,"name":"Womanizer - Tattoo","public_title":"Tattoo","sku":"172205168"},{"id":1509908217881,"price":14999,"name":"Womanizer - Blue","public_title":"Blue","sku":"1725205076"}]},"page":{"pageType":"product","resourceType":"product","resourceId":8993669708}};
for (var attr in meta) {
  window.ShopifyAnalytics.meta[attr] = meta[attr];
}</script>"""

soup = BeautifulSoup(s, "html.parser")
scr = soup.find("script")
m = re.search(r"var meta = (.*?);", scr.string)
if m:
    data = json.loads(m.group(1))
    for sku in data["product"]["variants"]:
        print(sku["sku"])

Output:

172145678
172146924
172150324
172205168
1725205076
user_5323120
Related