Not able to scrape <strong> tag properly using Beautifulsoup

user_14893468 0
Viewed 96

So I'm trying to scrape dates of the products from this adidas website using this code:

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/'
                         '84.0.4147.105 Safari/537.36'}
url = "https://www.adidas.com.sg/release-dates"
productsource = requests.get(url, headers=headers, timeout=15)
productinfo = BeautifulSoup(productsource.text, "lxml")


def jdMonitor():
    # webscraper
    all_items = productinfo.find_all(name="div", class_="gl-product-card")
    # print(all_items)
    for item in all_items:
        # print(item)
        pname = item.find(name="div", class_="plc-product-name___2cofu").text
        pprice = item.find(name="div", class_="gl-price-item").text
        imagelink = item.find(name="img")['src']
        plink = f"https://www.adidas.com.sg/{item.a['href']}"
        try:
            pdate = item.find(name="div", class_="plc-product-date___1zgO_").strong.text
        except AttributeError as e:
            print(e)
            pdate = "Data Not Available"
        print(f"""
        Product Name: {pname}
        Product Price: {pprice}
        Image Link: {imagelink}
        Product Link: {plink}
        Product Date: {pdate}
""")


jdMonitor()

But I'm getting an empty string in pdate. But if I use print(productinfo.find_all(name="strong")) to extract all the strong tags on the page, I'm able to extract all tags correctly just not the one I require. I'm getting the output as:

... <strong>All Recycled Materials</strong>, <strong> </strong> ...

The empty space between the second pair of strong tags should contain the dates like

<strong>Wednesday 30 Jun 21:30</strong>

Can someone explain why this is happening? and a way to extract it.

beautifulsoup python
1 Answers

It seems like the dates are dynamically updated and there are no such dates in the source code (open source code and look for "WEDNESDAY 30 JUN 19:00" and nothing will show up). The most obvious thing is to use selenium to make it work but it might be a slow solution. requests-html didn't work for me, the same thing just like with bs4. Rendering the page doesn't help either (or I was doing something wrong).

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
# running in headless mode for some reason gives no result or throws an error.
# options.headless = True 
driver = webdriver.Chrome(options=options)
driver.get('https://www.adidas.com.sg/release-dates')

for date in driver.find_elements_by_css_selector('.plc-product-date___1zgO_.gl-label.gl-label--m.gl-label--condensed'):
    print(date.text)
driver.quit()

# output:
'''
WEDNESDAY 30 JUN 19:00
THURSDAY 01 JUL 05:00
THURSDAY 01 JUL 05:00
THURSDAY 01 JUL 05:00
THURSDAY 01 JUL 05:00
THURSDAY 01 JUL 05:00
THURSDAY 01 JUL 05:00
THURSDAY 01 JUL 05:00
'''

You can also use regex to grab this dates (if they'll show up) like so:

import re

test = '''
Wednesday 30 Jun 19:00
THURSDAY 01 JUL 05:00
THURSDAY 01 FEb 25:00
'''
matches = re.findall(r"[a-zA-Z]+\s\d+\s\w+\s\d+:\d+", str(test))
finall_matches = '\n'.join(matches)
print(finall_matches)

# output before joining: "['Wednesday 30 Jun 19:00', 'THURSDAY 01 JUL 05:00', 'THURSDAY 01 FEb 25:00']"

# output after joining:
'''
Wednesday 30 Jun 19:00
THURSDAY 01 JUL 05:00
THURSDAY 01 FEb 25:00
'''
user_151646460
Related