Beautifulsoup Escaping Nested Tags

user_10400791 0
Viewed 37

I have the following HTML document,

html_part = '''
    <h3>First Header</h3>
    First Header Content
    <h3>Second Header</h3>
    Second Header with some annoying tag </br>
    which ruins the extraction </br>
    <h3>Third Header</h3>
    This works fine
'''

And I am trying to extract all the content between the h3 tags with the following program,

from bs4 import BeautifulSoup 


soup = BeautifulSoup(
    html_part,
    'html.parser'
)

for index in soup.find_all('h3'):
    print(
        index.next_sibling.string
    )
    print("-")

Which gives the following output,

First Header Content
-
Second Header with some annoying tag
-
This works fine
-

My desired output is the following,

First Header Content
-
Second Header with some annoying tag </br>
which ruins the extraction </br>
-
This works fine
-

How do I proceed?

beautifulsoup python
1 Answers

You could use soup.contents to work through all the elements and keep track of when each h3 tag is seen. For example:

from bs4 import BeautifulSoup
from bs4.element import Tag

html_part = '''
    <h3>First Header</h3>
    First Header Content
    <h3>Second Header</h3>
    Second Header with some annoying tag </br>
    which ruins the extraction </br>
    <h3>Third Header</h3>
    This works fine
'''

soup = BeautifulSoup(html_part, 'html.parser')
block = []
blocks = []
h3 = False

for el in soup.contents:
    if type(el) == Tag and el.name == 'h3':
        # Has a h3 tag been seen yet?
        if h3:
            blocks.append(block)
            block = []
        h3 = True
    elif h3:
        block.append(el)

# Add any final elements (missing a next h3)
if block:
    blocks.append(block)
    
for block in blocks:
    print(''.join(block))
    print('-')

This would give you:

    First Header Content
    
-

    Second Header with some annoying tag 
    which ruins the extraction 

-

    This works fine

-

Note: it is indented because your HTML is indented

If the content between the h3 tags is more complicated and needs to be kept (rather than just text), you could further process them as soup:

for block in blocks:
    soup.contents = block
    print(soup)
    print('-')
user_49857330
Related