Find next siblings until a certain one using beautifulsoup

user_1550725 0
Viewed 24678

The webpage is something like this:

<h2>section1</h2>
<p>article</p>
<p>article</p>
<p>article</p>

<h2>section2</h2>
<p>article</p>
<p>article</p>
<p>article</p>

How can I find each section with articles within them? That is, after finding h2, find nextsiblings

until the next h2.

If the webpage were like: (which is normally the case)

<div>
<h2>section1</h2>
<p>article</p>
<p>article</p>
<p>article</p>
</div>

<div>
<h2>section2</h2>
<p>article</p>
<p>article</p>
<p>article</p>
</div>

I can write codes like:

for section in soup.findAll('div'):
...
    for post in section.findAll('p')

But what should I do with the first webpage if I want to get the same result?

scrape siblings find beautifulsoup python
2 Answers

The next_siblings iterator can be helpful here as well:

for i in soup.find_all('h2'):
    for sib in i.next_siblings:
        if sib.name == 'p':
            print(sib.text)
        elif sib.name == 'h2':
            print ("*****")
            break
user_29210370
Related