How do I further narrow my search with BeautifulSoup?

user_15750261 0
Viewed 39

So I'm just learning how to scrape websites for data, and I have one I want to scrape to populate a database to do something with for practice. The site doesn't have all the information posted in one big page i can scrape, but instead has the information broken up into multiple "sets", with each set having its own page/collection of the data I want to scrape. All the "sets" are listed on a singular page though, with each set listed having the link to its individual page. I figured my best bet would be to scrape the "sets" page for their URL, and then request through to the "set" pages to collect the data I'm trying to get. Checking the html, each set is listed in a container, with the URL being the first thing listed within each section, like this:

<td class="flexbox">
<a href="url_i_need">
<more stuff i don't need>
</td>
<repeats_as_above_for_next_set>

what I've tried is:

response = requests.get('site_url')
content = response.content
soup = BeautifulSoup(content, 'html.parser')
data = soup.find_all('td', 'flexbox')

This seems to do the trick scraping each of the TD sections, but nothing I try let me further skim through the data to just the portion I need. After narrowing my search down to the general section I care about, how do I scrape the URL of each of those sections?

web-scraping beautifulsoup python
1 Answers

You can loop over data and for each item find nested tag a:

for item in data:
    link = item.find('a')
    url = link['href']

Btw is this line correct?

soup = BeautifulSoup.find_all(content, 'html.parser')

Standard way is this one:

soup = BeautifulSoup(content, 'html.parser')
user_29766120
Related