Trying to scrape elements within a div within a div, can't figure it out

user_13831283 0
Viewed 224

I am trying to use python to scrape the names of restaurants from a website. I'm having a hard time figuring out which exact div class to target and then how to write the code to do the scraping. I have successfully written the code for other webpages but can't figure it out for this one.

I am targetting this webpage: https://www.broadsheet.com.au/melbourne/fitzroy

Here is what I have tried:

soup_rest_list = BeautifulSoup(html_rest, 'html.parser')
type(soup_rest_list)

rest_container = soup_rest_list.find_all(class_="venue-teaser-list format-horizontal VenueTeaserListWrapper-sc-13dcca9-1 fIcGQi", "h2", class_="venue-title")

I'm not getting much love though. Right now when I execute my code I just get a []

Any help greatly appreciated.

web-scraping beautifulsoup python
2 Answers

Using find_all, you just have to look for tags h2 within class venue-title, and then extract its text attribute.

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup

options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')

driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)


url = 'https://www.broadsheet.com.au/melbourne/fitzroy'
driver.get(url)

page = BeautifulSoup(driver.page_source, 'html')
elements = page.find_all("h2", class_="venue-title")
names = [i.text for i in elements]

>> names
 
['Poodle Bar & Bistro',
 'Gogyo',
 'Rice Queen',
 'Vegie Bar',
 'Smith & Daughters',
 'Belles Hot Chicken Fitzroy',
 'Grub Fitzroy',
 'Archie’s All Day',
 'Sonido',
 'Gabriel',
 'Mile End Bagels',
 'Napier Quarter',
 'Bonny',
 'Near & Far',
 'The Everleigh',
 "Milney's",
 'Mono-XO',
 'The Rum Diary Bar',
 'Smith & Deli',
 'Meatsmith Fitzroy',
 'American Vintage',
 'Hunter Gatherer',
 'Plane',
 'Aesop']
user_110978660

First off, if you actually tried what you tried, i.e.

rest_container = soup_rest_list.find_all(class_="venue-teaser-list format-horizontal VenueTeaserListWrapper-sc-13dcca9-1 fIcGQi", "h2", class_="venue-title")

Python would have reported a syntax error rather than assigning [] to rest_container, as 1) "h2" is a positional argument that came after class_, and then 2) class_ was specified a second time as a keyword argument.

What you are probably looking for is the CSS selector functionality, which will let you select the elements within a set of elements like you wanted by specifying the equivalent CSS selector rule:

>>> soup_rest_list.select("div.venue-teaser-list.format-horizontal.VenueTeaserListWrapper-sc-13dcca9-1.fIcGQi h2.venue-title")
[<h2 class="venue-title">...]
user_29048960
Related