Filtering subclass <p> tags with Beautiful Soup

user_15309078 0
Viewed 178

I am trying to grab the text on a webpage which is organized into paragraphs with <p> tags including a subclass that also contains <p> tags. When I use find_all, it grabs the tags I want as well as the ones within the subclasses. I want to exclude the subclass tags.

Here is the website's html:

> div id="storytext" class="class_id"
    > div id="imageid" class="imagewrap"
        > # here is the subclass <p> tags and are being scraped when I use find_all()
    # the following that is the only <p> tag I want to include
    > <p>...</p>
    > <p>...</p>
    > <p>...</p>
    > <p>...</p>
    > <p>...</p>

My code:

story_text = soup.find('div', {'id':'storytext'})
paragraphs = story_text.find_all('p')
for p in paragraphs:
    story.append(p.text)

This pulls everything under the div, {'class':'imagewrap'}

web-scraping beautifulsoup
3 Answers

You can .extract the tag with <p> from the soup and then select all other <p> tags:

from bs4 import BeautifulSoup


html_doc = """
<div id="storytext" class="class_id">
    <div id="imageid" class="imagewrap">
        <p> here is the subclass tags and are being scraped when I use find_all()</p>
    </div>
   <p>1..</p>
   <p>2..</p>
   <p>3..</p>
   <p>4..</p>
   <p>5..</p>
</div>
"""

soup = BeautifulSoup(html_doc, "html.parser")

story_text = soup.select_one("#storytext")

# remove the #imageid tag
imageid = story_text.select_one("#imageid")
imageid.extract()

# select all <p> tags
for p in story_text.select("p"):
    print(p.text)

Prints:

1..
2..
3..
4..
5..
user_100359850 
from bs4 import BeautifulSoup

html_doc = """
<div id="storytext" class="class_id">
    <div id="imageid" class="imagewrap">
        <p> here is the subclass tags and are being scraped when I use find_all()</p>
    </div>
   <p>1..</p>
   <p>2..</p>
   <p>3..</p>
   <p>4..</p>
   <p>5..</p>
</div>
"""

soup = BeautifulSoup(html_doc, 'lxml')
print([x.text for x in soup.select('#imageid ~ p')])

Output:

['1..', '2..', '3..', '4..', '5..']
user_76589850

You can use the CSS selector "#storytext > p" to grab only immediate p children of #storytext:

>>> from bs4 import BeautifulSoup
>>> html = """
... <div id="storytext" class="class_id">
...     <div id="imageid" class="imagewrap">
...         <p> here is the subclass tags and are being scraped when I use find_all()</p>
...     </div>
...    <p>1..</p>
...    <p>2..</p>
...    <p>3..</p>
...    <p>4..</p>
...    <p>5..</p>
... </div>"""
>>> soup = BeautifulSoup(html, "lxml")
>>> soup.select("#storytext > p")
[<p>1..</p>, <p>2..</p>, <p>3..</p>, <p>4..</p>, <p>5..</p>]
>>> [x.text for x in soup.select("#storytext > p")]
['1..', '2..', '3..', '4..', '5..']

This answer does the opposite, using ~ instead of >. Which makes more sense to use depends on context for the HTML.

user_62433520
Related