I am currently scraping this website to build a car dataset and I have an equation built to loop through each page of the website while scraping. However, I am unable to extract the text I need to make this work.
The below code snippet is the tag that I am trying to scrape. I need to get the number of vehicles on the site.
<span class="d-none d-sm-inline">166 Vehicles</span>
This image shows the site's element that I am trying to scrape
Below is my code that I am using to scrape that element:
# Packages
import pandas as pd
import numpy as np
from bs4 import BeautifulSoup
import requests
print("Started web scrape...")
limit = 10
start = 0 #increment by limit
website = requests.get(f'https://www.sosubaru.com/new-inventory/index.htm?start={start}')
soup = BeautifulSoup(website.text, 'html.parser')
inventory_count = soup.select("span.d-none.d-sm-inline")[0].string
print(inventory_count)
This code returns the following:
Started OR_GP_Roe_Motors web scrape...
Traceback (most recent call last):
File "c:/mypath...", line 16, in <module>
inventory_count = soup.select("span.d-none.d-sm-inline")[0].string
IndexError: list index out of range
Then I checked to see why I was getting that error code by returning everything that the soup.select gave me:
inventory_count = soup.select("span.d-none.d-sm-inline")
print(inventory_count)
which returned:
Started web scrape...
[]
Why is it giving me an empty list?
I then told it to print out every span tag on the website to see if it was there. The result printed out many span tags but didn't include the one I am looking for. Why can't I detect it with beautiful soup? Is it the parser I am using? I tried using 'lxml' as the parser but it didn't change anything. Does it have anything to do with the fact that the website is an html xmls doc?
I have already scraped a few websites and haven't had any problems like this until now.