Web scraping - Text in tag not achievable

Question

I have been trying to scrape this page (https://www.lojasrenner.com.br/lista/promocoes/-/N-1hwylc0Z1l95pyv/p1?s_icid=20191226_HOME_MENU_PROMO_SALDOS_DESK) with selenium with Chrome driver, that contains all products with discount. The scraper detects all the product information (including discounted price) except for the discount (present in the flag discount, or in the product information as old price - were I could calculate the discount with the two prices) [Inspect of the discount flag][1] [Inspect of the striked price][2].

Somehow, it randomly gives me a number of the discounts, but I can't achieve consistenly to get all the discounts of the page (there should be 48).I've used explicit and implicit waits (2 minutes waits, expected condition that the element contains text). I don't know what the problem is. Here is my code. Any ideas what's going on? [1]: https://i.stack.imgur.com/hLqCK.png [2]: https://i.stack.imgur.com/noV76.png

import ast
from selenium import webdriver
from bs4 import BeautifulSoup
quotes = []
driver = webdriver.Chrome(PATH)
URL = 'https://www.lojasrenner.com.br/lista/promocoes/-/N-1hwylc0Z1l95pyv/p1?s_icid=20191226_HOME_MENU_PROMO_SALDOS_DESK'
driver.get(URL)
delay = 120 # secs
#time.sleep(delay)
html = driver.page_source.encode('utf-8')
soup = BeautifulSoup(html)
driver.quit()
content = soup.findAll("div", {"class": re.compile(r"item_product animation-element bounce-up cf")})
if content == []:
    break
for row in content:
    b = row.get("data-product-gtm")
    dic = ast.literal_eval(b)
    del dic['list'] ; del dic['position']
    c = row.find("span", {"class": "flag_discount"})
    if c != None:
        dic['discount'] = c.text
    else:
        dic['discount'] = ''
    quotes.append(dic)

user_1178971 · Answer

The web site uses service workers to push additional items to the page as you scroll down. You would need to have JS enabled and use it to scroll down the page so that the additional JS push requests were triggered and the new products appeared.

An alternative solution would be to pull the initial products out of the doc dataLayer. It will give you the initial 21 results.

You could then inspect the JavaScript on the page to create your own requests to the API. They take this form. This method is faster than running web drivers and if you work out the mechanisms of the web site then you can presumably request all of the products.

To do this, I would recommend using requests and parsing the content with beautiful soup.

You can get the data layer out of the script as follows:

from bs4 import BeautifulSoup
import requests
import json
session = requests.Session()
r = session.get(url=URL)
soup = BeautifulSoup(r.content, 'html.parser')
all_scripts = soup.find_all('script')
content_dict = extract_json_from_script(all_scripts, 'var dataLayer = [];dataLayer.push(', ')')

Using this function:

def extract_json_from_script(all_scripts, start_split, end_split, trim_length=None):
    parts = []
    for script in all_scripts:
        try:
            scriptstr = str(script)
            parts = scriptstr.split(start_split)
        except ValueError as e:
            print(e)
            pass
        if len(parts) > 1:
            break
    jsonstr = parts[1].split(end_split)[0].strip()
    if trim_length is not None:
        jsonstr = jsonstr[:trim_length]
    jsondict = json.loads(jsonstr)
    return jsondict

user_10035985 · Answer

The data of products are loaded dynamically. You can use their API to obtain the information about price, discount, images etc.

For example:

import json
import requests
from bs4 import BeautifulSoup


url = "https://www.lojasrenner.com.br/lista/promocoes/-/N-1hwylc0Z1l95pyv/p1?s_icid=20191226_HOME_MENU_PROMO_SALDOS_DESK"
u = 'https://www.lojasrenner.com.br/rest/model/lrsa/api/CatalogActor/productBoxDataDesk'
params = {
    'pushSite': "rennerBrasilDesktop",
    'productId':   "548885748",
    'skuId':   "550553478",
    'skuList': "550553460,550553478",
    'skuGift': "",
    'stampIds':    "",
    'groupOneId':  "color",
    'findCategories':  "false"
}
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

for p in soup.select('[data-product]'):
    data = json.loads(p['data-product'])
    params['productId'] = data['productId']
    params['skuId'] = data['skuId']
    params['skuList'] = ','.join(data['skuList'])
    product_data = requests.get(u, params=params).json()

    #uncomment this to print all data:
    # print(json.dumps(product_data, indent=4))

    print('{:<50}{:<10}{:<10}{:<10}'.format(product_data['displayName'], product_data['listPriceFormatted'], product_data['salePrice'], product_data['percentDiscount']))

Prints:

Blusa Lisa Gola V em Linho                        R$ 79,90  59.9      25        
Macacão Liso em Crepe com Cinto                   R$ 159,90 99.9      37        
Blusa Lisa em Fake Suede                          R$ 79,90  59.9      25        
Camisa Manga Longa Lisa com Decote V              R$ 99,90  59.9      40        
Blusa Lisa em Fake Suede                          R$ 79,90  59.9      25        
Camisa Estampada com Botões e Amarração           R$ 99,90  59.9      40        
Lenço Leve Estampa Animal Print                   R$ 69,90  49.9      28        
Blusa Bata Estampa Poá                            R$ 79,90  59.9      25        
Calça Legging Lisa com Detalhe de Material Sintético nos BolsosR$ 139,90 79.9      42        
Blusa Manga Curta Bufante Lisa                    R$ 99,90  59.9      40        
Macacão Estampado Folhagens                       R$ 139,90 99.9      28        
Short Estampado com Coqueiros em Sarja            R$ 89,90  39.9      55        
Blusa Lisa Gola V em Linho                        R$ 79,90  59.9      25        
Blusa Estampada Flor                              R$ 69,90  39.9      42        
Blusa com Estampa Hakuna Matata                   R$ 39,90  29.9      25        
Blusa Sem Manga Lisa Gola Altinha                 R$ 69,90  39.9      42        
Calça Legging Xadrez com Bolso Traseiro           R$ 99,90  59.9      40        
Blusa Lisa em Fake Suede                          R$ 79,90  59.9      25        
Vestido Liso em Fake Suede                        R$ 119,90 79.9      33        
Blusa Manga Longa Gola Alta em Tule               R$ 49,90  39.9      20        
Blusa Manga Curta Estampa Folhagem com Detalhe    R$ 69,90  39.9      42        
Blusa Estampada com Folhagens Manga Longa         R$ 79,90  59.9      25        
Lenço Leve Estampa Cobras e Bloco de Cor          R$ 69,90  49.9      28        
Vestido em Jersey Listrado com Abertura no Ombro  R$ 139,90 59.9      57        
Blusa Manga Longa Estampa Folhagem com Botões     R$ 99,90  59.9      40        
Regata Estampada com Corda                        R$ 59,90  39.9      33        
Blusa Lisa Gola V em Linho                        R$ 79,90  59.9      25        
Saia com Estampa Folhagens em Sarja               R$ 139,90 59.9      57        
Regata Floral com Amarração                       R$ 99,90  59.9      40        
Blusa Manga Longa Listrada Transpassada           R$ 79,90  59.9      25        
Saia Midi Floral com Babados                      R$ 159,90 99.9      37        
Vestido Midi Manga Curta Decote V Estampa Folhagem com Cinto CordaR$ 139,90 79.9      42        
Macacão Liso Midi Manga Longa com Puxadinho       R$ 119,90 79.9      33        
Blusa Animal Print                                R$ 89,90  59.9      33        
Blusa com Detlhe no Ombro em Crepe                R$ 79,90  39.9      50        
Saia Longa Estampada com Babados                  R$ 159,90 99.9      37        
Vestido Liso em Fake Suede                        R$ 119,90 79.9      33        
Calça Listrada com Amarração                      R$ 139,90 99.9      28        
Blusa Estampada com Coqueiros                     R$ 79,90  59.9      25        
Blusa Lisa com Aplicação nas Mangas               R$ 89,90  39.9      55        
Blusa Lisa com Zíper nas Costas                   R$ 89,90  39.9      55        
Blusa com Bordado Babados                         R$ 89,90  59.9      33        
Calça Legging Pied de Poule                       R$ 99,90  59.9      40        
Calça Legging Esportiva com Recorte               R$ 89,90  79.9      11        
Saia Midi Floral com Fivela                       R$ 159,90 99.9      37        
Cardigan Básico Alongado com Bolsos               R$ 89,90  59.9      33        
Vestido Básico em Ribana Listrado                 R$ 59,90  39.9      33        
Blusa Lisa em Tule Curve & Plus Size              R$ 99,90  79.9      20

Web scraping - Text in tag not achievable

2 Answers