Pages take 2 loadings to complete

Question

On this website, https://toptees.store/linux-funny-cloud-computing I try to scrape sold span text but this website takes 2 time load to coming complete website. That's why data is not scraped.

My Code:

import requests
from bs4 import BeautifulSoup
url = "https://toptees.store/linux-funny-cloud-computing"

reqs = requests.get(url)
soup = BeautifulSoup(reqs.text, 'lxml')
sold = soup.find_all("span", class_='ng-binding')
print(sold)

I also tried with selenium with Beautifulsoup

import time
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))  # ,options=options

filepath = 'urls.txt'
with open(filepath) as f:
    urls = [i.strip() for i in f.readlines()]

titles = []
for url in urls:
    driver.get(url)
    driver.maximize_window()
    time.sleep(3)
    soup = BeautifulSoup(driver.page_source, 'lxml')
    sold = soup.find('span', class_="ng-binding")
    print(sold)

The output is coming like this [] . How can I scrape this link with Beautifulsoup?

Barry the Platipus · Answer

This is one way to get that information you're after:

import time as t
from bs4 import BeautifulSoup as bs
import undetected_chromedriver as uc


options = uc.ChromeOptions()
options.add_argument("--no-sandbox")
options.add_argument('--disable-notifications')
options.add_argument("--window-size=1280,720")
# options.add_argument('--headless')

browser = uc.Chrome(options=options)
url = 'https://toptees.store/linux-funny-cloud-computing'

browser.get(url)
t.sleep(1)
browser.get(url)
t.sleep(7)
soup = bs(browser.page_source, 'html.parser')
sold = soup.select_one('days-available[any-sold="campaign.sold"]')
title = soup.select_one('h1.campaign-name-title')
print(title.text, '|', [x for x in sold.text.split(' ') if len(x.strip()) > 0][0])

Result printed in terminal:

Linux funny Cloud Computing | 43

For undetected_chromedriver, please see https://pypi.org/project/undetected-chromedriver/ [instructions on how to set it up, etc]

Pages take 2 loadings to complete

1 Answers