Extract all links after a particular tag using beautifulsoup

Admin 1
Viewed 208

I couldn't find an answer to my specific question on SO so here it goes. Suppose I have an HTML that looks something like (sorry if it looks a bit stupid, I just wanted to make a very minimal example that captures my problem):

html1 = """<html>
<head></head>
<body>
<p>Hello World!</p>
<a href='whatevs.com'>whatevs</a>
<p>Howdy!</p>
<a href='well.com'>well</a>
<div><span>haha</span><a href='haha.com'>haha</a></div>
<a href='goodbye.com'>Goodbye!</a>
</body>
</html>"""

I want to extract all the links in here that come after <p>Howdy!</p>.

So I tried,

howdy = BeautifulSoup(html1).find('p', text='Howdy!')

but both howdy.find_next_siblings('a') and howdy.find_next('a') return something a bit different from what I want.

Desired output:

[<a href="well.com">well</a>,
 <a href="haha.com">haha</a>,
 <a href="goodbye.com">Goodbye!</a>]

(Well actually, it's ['well','haha','Goodbye!'] but I think I can get this from above.)

Ideally, I want something like howdy.find_all('a').

Sorry if it's too basic a question but I'd really appreciate if anyone can direct to the right direction. Thanks!

beautifulsoup python-3.x html
2 Answers

Try:

from bs4 import BeautifulSoup

html1 = """<html>
<head></head>
<body>
<p>Hello World!</p>
<a href='whatevs.com'>whatevs</a>
<p>Howdy!</p>
<a href='well.com'>well</a>
<div><span>haha</span><a href='haha.com'>haha</a></div>
<a href='goodbye.com'>Goodbye!</a>
</body>
</html>"""

soup = BeautifulSoup(html1, "html.parser")

out, tag = [], soup.find("p", text="Howdy!")
while True:
    tag = tag.find_next("a")
    if not tag:
        break
    out.append(tag.text)

print(out)

Prints:

['well', 'haha', 'Goodbye!']
user_100359850

.find_next_siblings() is what you are looking for:

from bs4 import BeautifulSoup

html1 = """<html>
<head></head>
<body>
<p>Hello World!</p>
<a href='whatevs.com'>whatevs</a>
<p>Howdy!</p>
<a href='well.com'>well</a>
<div><span>haha</span><a href='haha.com'>haha</a></div>
<a href='goodbye.com'>Goodbye!</a>
</body>
</html>"""

soup = BeautifulSoup(html1, 'lxml')
siblings = soup.find('p', text="Howdy!").find_next_siblings()
a_tags = [sib.find('a').text if sib.name != 'a' else sib.text for sib in siblings]
# -> ['well', 'haha', 'Goodbye!']

Note: Using .find_next_siblings('a') would not give the desired result since it doesn't look for inner tags.

user_119235580
Related