Scrape everything between two unested tags

Admin 1
Viewed 41

Is it possible to scrape everything between two unested tags ?

For instance:

<h3>Title 1<h3>
<div class="div">
    <span class="span">span1</span>
    <label class="label">label1</label>
</div>
<div class="div">
    <span class="span">span2</span>
</div>
<h3>Title 2<h3>
<div class="div">
    <span class="span">span3</span>
    <label class="label">label2</label>
</div>
<div id="div">
    <span id="span">span4</span>
</div>

So I would like to scrape just what is located under Title 1 until Title 2. Is this possible using bs4 ?

Right now I have something like this (problem is it scrape everything since classes are all the same):

   for i in soup.findAll("div",{"class":"div"}):
       print(i.span.text)

Now I get:

span1
span2
span3
span4

I'd like to get:

span1
span2
beautifulsoup html python
2 Answers

I don't know if this is best solution to this problem but you can split your text and scrape only the part that you need.

text = """
<h3>Title 1</h3>
<div class="div">
    <span class="span">span1</span>
    <label class="label">label1</label>
</div>
<div class="div">
    <span class="span">span2</span>
</div>
<h3>Title 2</h3>
<div class="div">
    <span class="span">span3</span>
    <label class="label">label2</label>
</div>
<div id="div">
    <span id="span">span4</span>
</div>
"""

sub_text = text.split(soup.find('h3', text="Title 2").string)[0]

This will give:

'"\n<h3>Title 1</h3>\n<div class="div">\n    <span class="span">span1</span>\n    <label class="label">label1</label>\n</div>\n<div class="div">\n    <span class="span">span2</span>\n</div>\n<h3>'

After converting that string into a bs4 object, you can scrape all you need:

scrape_me = BeautifulSoup(sub_text, 'lxml')

for i in scrape_me.findAll("div", class_="div"):
    print(i.span.text)
# -> span1 span2
user_119235580

One approach is:

  1. find the second class="span", then navigate backwards, and find_all_previous() div.

  2. The tags are in backward order, so use the reversed() function..

  3. find the <span> tags

from bs4 import BeautifulSoup

html = """
<h3>Title 1<h3>
<div class="div">
    <span class="span">span1</span>
    <label class="label">label1</label>
</div>
<div class="div">
    <span class="span">span2</span>
</div>
<h3>Title 2<h3>
<div class="div">
    <span class="span">span3</span>
    <label class="label">label2</label>
</div>
<div id="div">
    <span id="span">span4</span>
</div>
"""
soup = BeautifulSoup(html, "lxml")

for tag in reversed(
    soup.select_one("div:nth-of-type(2) span.span").find_all_previous("div")
):
    print(tag.find("span").text)

Output:

span1
span2
user_123497340
Related