Beautiful soup: Extract everything between two tags

user_14139172 0
Viewed 921

I am using BeautifulSoup to extract data from HTML files. I want to get all of the information between two tags. This means that if I have an HTML section like this:

<h1></h1>
Text <i>here</i> has no tag
<div>This is in a div</div>
<h1></h1>

Then if I wanted all of the information between the first h1 and the second h1, the output would look like this:

Text <i>here</i> has no tag
<div>This is in a div</div>

I've tried nextsibling loops, but there always seems to be a catch. Is there a command in beautifulsoup that simply pulls everything (Text, newlines, divs, special characters) that is between element "A" and element "B"?

beautifulsoup html python
2 Answers

One solution is to .extract() all content in front of first <h1> and after second <h1> tag:

from bs4 import BeautifulSoup


html_doc = '''
This I <b>don't</b> want
<h1></h1>
Text <i>here</i> has no tag
<div>This is in a div</div>
<h1></h1>
This I <b>don't</b> want too
'''

soup = BeautifulSoup(html_doc, 'html.parser')

for c in list(soup.contents):
    if c is soup.h1 or c.find_previous('h1') is soup.h1:
        continue
    c.extract()

for h1 in soup.select('h1'):
    h1.extract()

print(soup)

Prints:

Text <i>here</i> has no tag
<div>This is in a div</div>
user_100359850

Here is how, you can simply target their parent or you can wrap them in the container and extract all children of that parent you're targeting like the following

from bs4 import BeautifulSoup
content = """
<div class="container">
    <h1></h1>
        Text <i>here</i> has no tag
        <div>This is in a div</div>
    <h1></h1>
</div>
"""
soup = BeautifulSoup(content, 'html.parser')
results = soup.find('div').findChildren()
print(results)

or

print(soup.find('h1').findAllNext())
user_135368930
Related