How could I extract a specific numerical value from a list item

user_16939735 0
Viewed 23

So I'm trying to scrape data from a website and get specific values that I will use later in calculations but I am having trouble taking the data I scrape and pulling just the values I want from it. Currently, this is what I have:

import requests
from bs4 import BeautifulSoup

header = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux i686 on x86_64)'
}
url = 'https://cars.usnews.com/cars-trucks/ram/1500/2021/specs/1500-tradesman-4x2-quad-cab-6-4-box-414114'

page = requests.get(url, headers=header) # change headers or get blocked
soup = BeautifulSoup(page.content, 'html.parser')
specs = soup.find_all('div', class_="trim-specs columns small-12")
spec_values = []

for spec in specs:
    spec_values.extend(spec.find_all('li'))

towing = [x for x in spec_values if 'Maximum Trailering Capacity (lbs.)' in x.string]

print(towing)

From here I get this output: [<li>Maximum Trailering Capacity (lbs.): 7730</li>]

How could I just pull the value of 7330 from here?

This is one way I found of doing this but it won't work for values that are not integers

towing_num = [int(i) for i in str(towing) if i.isdigit()]
towing_cap = int(''.join(map(str, towing_num)))
print(towing_cap)

This gives me 7730 as an output but this method does not work for any number with a decimal. Is there a more straightforward way of obtaining this value?

Thanks in advance

web-scraping python-3.x
1 Answers

Looking at the page, you can split the spec with : and then second element is your number. You can then apply int() or float() on it:

import requests
from bs4 import BeautifulSoup

header = {"User-Agent": "Mozilla/5.0 (X11; Linux i686 on x86_64)"}
url = "https://cars.usnews.com/cars-trucks/ram/1500/2021/specs/1500-tradesman-4x2-quad-cab-6-4-box-414114"

page = requests.get(url, headers=header)  # change headers or get blocked
soup = BeautifulSoup(page.content, "html.parser")

# load all specs into `specs` list
specs = []
for li in soup.select(".trim-specs li:not(.sub-header)"):
    specs.append([w.strip() for w in li.text.split(":")])

# find "Maximum Trailering Capacity (lbs.)" in specs:
for s in specs:
    if "Maximum Trailering Capacity (lbs.)" in s:
        print("{} is {}".format(s[0], int(s[1])))
        break

Prints:

Maximum Trailering Capacity (lbs.) is 7730
user_100359850
Related