How do I grab the text inside of a span tag in Python using Beautiful Soup

user_9318827 0
Viewed 76

I'm having a devil of a time trying to get the text from a list of items on petango.com. Screenshot of html below I load the page using selenium to let it load. then grab it for a bs object

PetSoup = BeautifulSoup(page_source, 'html.parser')

I've tried:

ul_tag = PetSoup.findAll('div', {'class': 'group details-list'})
pet_details = [i.get_text() for i in ul_tag]
print(pet_details)

this gets me all of the items as one list object (I think) but I can't seem to parse it out so the I can assign variables to each item (e.g. Age:, Breed:, etc.)

['\n\nBreed: Chihuahua, Short Coat / Mix\nAge: 15y 5m Gender: Male\nColor: Black / Grey\nSpayed/Neutered: Yes\nSize: Small\nDeclawed: No\nAdoption Date: \n\n']

I've also tried:

for ultag in PetSoup.find_all('ul', {'class': 'group details-list'}):
    for litag in ultag.find_all('li'):
        print(litag.text)

which gets me all of the text, but again, can't seem to parse it to be able to assign variables

What I really want to do is just grab the text in the span tag but I can't seem to grab it. I think it has something to do with how the span tag is structured, but I can't seem to get it by trying different variations of items in the span. I just get an empty return. The specific span is:

<span data-bind="text: breed">Terrier, Rat / Mix </span>

Can anyone point me in the right direction? Thank you for your help! the specific page is here: https://www.petango.com/Adopt/Dog-Terrier-Rat-22192827

enter image description here

web-scraping selenium beautifulsoup python
2 Answers

To extract text from the span tag, simply use span.text. Here is how I extracted the span text:

from bs4 import BeautifulSoup

html = '<span data-bind="text: breed">Terrier, Rat / Mix </span>'

soup = BeautifulSoup(html,'html.parser')

span = soup.find('span')

print(span.text)

Output:

Terrier, Rat / Mix

This works fine. But I went a step further to scrape all the data from the website. Here is the full code to do it:

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Chrome()

driver.get('https://www.petango.com/Adopt/Dog-Terrier-Rat-22192827')

html = driver.page_source

driver.close()

soup = BeautifulSoup(html,'html.parser')

ul = soup.find('div',class_ = 'group details-list').ul #Gets the ul tag

li_items = ul.find_all('li') #Finds all the li tags within the ul tag

headings = []
values = []

for li in li_items:
    heading = li.strong.text
    headings.append(heading)
    
    value = li.span.text
    
    if value:
        values.append(value)
    else:
        values.append(None)

U can also create a beautiful Pandas DataFrame with these lists (for better readability) by adding these lines to the code:

details_dict = {'Headings':headings,
                'Values':values}

df = pd.DataFrame(details_dict)

print(df)

Output:

       Headings              Values
0            Breed:  Terrier, Rat / Mix
1              Age:              11y 7m
2            Color:       White / Black
3  Spayed/Neutered:                 Yes
4             Size:               Small
5         Declawed:                  No
6    Adoption Date:                None

Hope that this helps!

user_123098080

Solution without selenium:

import re
import json
import requests 
from bs4 import BeautifulSoup


url = 'https://www.petango.com/Adopt/Dog-Terrier-Rat-22192827'
ajax_url = 'https://www.petango.com/DesktopModules/Pethealth.Petango/Pethealth.Petango.DnnModules.AnimalDetails/API/Main/GetAnimalDetails?moduleId={}&animalId={}&clientZip=null'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0',}

html_text = requests.get(url, headers=headers).text
soup = BeautifulSoup(html_text, 'html.parser')

animal_id = url.split('-')[-1]
module_id = soup.select_one('[id^="Module-"]:has(.pet-id)')['id'].split('-')[-1]
tab_id = re.search(r'`sf_tabId`:`(\d+)`', html_text).group(1)

headers['TabId'] = tab_id
data = json.loads( requests.get(ajax_url.format(module_id, animal_id), headers=headers).text ) 

# print data to screen:
print(json.dumps(data, indent=4))

Prints:

{
    "id": 22192827,
    "speciesId": 1,
    "species": "Dog",
    "name": "Charlie 3",
    "photo": "https://g.petango.com/photos/1488/6955fc68-7a2e-4da1-af34-930c5fabd713.jpg",
    "photo2": "https://g.petango.com/photos/1488/450f6d7d-e501-475c-9b1f-9322dbc225eb.jpg",
    "photo3": "https://g.petango.com/photos/1488/4fbdbf1e-58b0-45c8-8c12-b5122ad1fff8.jpg",
    "youtubeVideoId": null,
    "age": "11y 7m",
    "distance": 0,
    "gender": "Male",
    "breed": "Terrier, Rat / Mix",
    "color": "White / Black",
    "spayedNeutered": "Yes",
    "size": "Small",
    "memo": null,
    "noDogs": false,
    "noCats": false,
    "noKids": false,
    "addToFavorites": false,
    "removeFromFavorites": false,
    "adoptionDate": null,
    "declawed": false,
    "adoptionApplicationUrl": "https://www.petango.com/Adoption-Application?shelterId=1897&animalId=22192827",
    "shelterName": "OH",
    "shelterCityState": "CINCINNATI, OH"
}
user_100359850
Related