Get the descriptions of product details in the script by BeautifulSoup and json

user_15002355 0
Viewed 69

I'm getting descriptions of product details in the tag from the web.

Here's the code :

import re
import json
from bs4 import BeautifulSoup
import requests

url  = 'https://oldnavy.gap.com/browse/product.do?pid=599211032&rrec=true&mlink=5050,12413545,onproduct1_rr_3&clink=12413545#pdp-page-content'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
results = soup.find('script', id="pdpData").string
jsData = re.search(r'window.__ PRODUCT_PAGE_STATE __\s+=\s+', results)
data = json.loads(jsData.group(0))

and the part xxxxx I want is below in the script

window. __PRODUCT_PAGE_STATE __ = JSON.parse(xxxxx)

Through re.search to find window. __PRODUCT_PAGE_STATE __ , I still can not reach the part xxxxx.

Is there any other way to extract the info in the part xxxxx?

python-re beautifulsoup python
1 Answers

Try this:

import re
import json
from bs4 import BeautifulSoup
import requests

url = 'https://oldnavy.gap.com/browse/product.do?pid=599211032&rrec=true&mlink' \
      '=5050,12413545,onproduct1_rr_3&clink=12413545#pdp-page-content'

soup = BeautifulSoup(
    requests.get(url).content,
    "html.parser",
).find('script', id="pdpData")

the_xxx_part = json.loads(
    re.search(r"\.parse\((.+)\);", soup.string, re.S).group(1).strip(),
)
print(json.loads(the_xxx_part)["productData"]["name"])

Output:

Unisex Faux-Fur-Trim Hooded Frost-Free Puffer Jacket for Toddler

To print the entire JOSN object edit this:

print(json.loads(the_xxx_part)["productData"]["name"])

into this:

print(json.loads(the_xxx_part))
user_61067910
Related