Output from web scraping with bs4 returns empty lists

Chioma Amuwa 0
Viewed 47

I am trying to scrape specific information from a website of 25 pages but when I run my code i get empty lists. My output is supposed to be dictionary with the specific information scraped. Please any help would be appreciated.

# Loading libraries
import requests
from bs4 import BeautifulSoup
import pandas as pd
import mitosheet

# Assigning column names using class_ names
name_selector = "af885_1iPzH"
old_price_selector = "f6eb3_1MyTu"
new_price_selector = "d7c0f_sJAqi"
discount_selector = "._6c244_q2qap"

# Placeholder list
data = []

# Looping over each page
for i in range(1,26):
   url = "https://www.konga.com/category/phones-tablets-5294?brand=Samsung&page=" +str(i)
   website = requests.get(url)
   soup = BeautifulSoup(website.content, 'html.parser')

   name = soup.select(name_selector)
   old_price = soup.select(old_price_selector)
   new_price = soup.select(new_price_selector)
   discount = soup.select(discount_selector)

   # Combining the elements into a zipped list to be able to pull the data simultaneously
   for names, old_prices, new_prices, discounts in zip(name, old_price, new_price, discount):
      dic = {"Phone Names": names.getText(),"New Prices": new_prices.getText(),"Old Prices": old_prices.getText(),"Discounts": discounts.getText()}
      data.append(dic)

   data
beautifulsoup web-scraping python-3.x
1 Answers

I tested the below and it works for me getting 40 name values.

I wasn't able to get the values using beautiful soup but directly through selenium.

If you decide to use Chrome and PyCharm as I have then:

Open Chrome. Click on three dots near top right. Click on Settings then About Chrome to see the version of your Chrome. Download the corresponding driver here. Save the driver in the PyCharm PATH folder

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

# Assigning column names using class_ names
name_selector = "af885_1iPzH"

# Looping over each page
for i in range(1, 27):
    url = "https://www.konga.com/category/phones-tablets-5294?brand=Samsung&page=" +str(i)
    driver.get(url)

    xPath = './/*[@class="' + name_selector + '"]'
    name = driver.find_elements(By.XPATH, xPath)
Cyrus0
Related