I can't optimise a beautifullsoup query with FIND_NEXT_SIBLING instead of PARENT and NEXT_ELEMENT

user_8614742 0
Viewed 54

I am trying to get a video link (on vimeo.com) using beautifulsoup. I managed to get what I wanted with the code below, but I'm trying to make it simpler and I can't...

HTML SOURCE:

<h5 class="m-b-5"><b>Devoirs </b>
</h5>
<p class="text-muted m-b-30 font-13">
    N/A </p>
<h5 class="m-b-5"><b>Commentaires sur la performance </b></h5>
<p class="text-muted m-b-30 font-13">
    Today finished up the writing and completed the listening assessment. Next week she'll complete the speaking and be
    finished! </p>

<h5 class="m-b-5"><b>Lien vers la vidéo </b></h5>
<p class="text-muted m-b-30 font-13">
    <a href="https://vimeo.com/zxzzxzx" target="_blank">https://vimeo.com/zxzxzx</a>
</p>

PYTHON CODE:

soup = BeautifulSoup(Session1.text, "lxml")
regex = re.compile('Lien vers la vidéo')
vimeolink=soup.find(string=regex).parent.parent.next_sibling.next_element.next_element.next_element.next_element
# tried the following but it's not working:
# vimeolink=soup.find(string=regex).parent('h5').find_next_sibling('a)

print(vimeolink)

result:
https://vimeo.com/zxzzxz

I tried with find_next_sibling but without success.

web-scraping beautifulsoup optimization python
2 Answers

Try:

a = soup.select_one('h5:-soup-contains("Lien vers la vidéo")').find_next("a")["href"]
print(a)

Prints:

https://vimeo.com/zxzzxzx

Or:

a = soup.select_one('h5:-soup-contains("Lien vers la vidéo") + p a')["href"]
print(a)

Note: If you use older version of bs4, change :-soup-contains to :contains

user_100359850

To get the correct link, you can use a CSS selector: p.text-muted.m-b-30.font-13:has(a) a which will select a p tag with the class text-muted m-b-30 font-13 that as an a. The other tags with the same class don't have an a, so, search for the one that does, which includes your link

To use a CSS selector, use the select_one() method instead of .find().

from bs4 import BeautifulSoup

html = """<h5 class="m-b-5"><b>Devoirs </b>
</h5>
<p class="text-muted m-b-30 font-13">
    N/A </p>
<h5 class="m-b-5"><b>Commentaires sur la performance </b></h5>
<p class="text-muted m-b-30 font-13">
    Today finished up the writing and completed the listening assessment. Next week she'll complete the speaking and be
    finished! </p>

<h5 class="m-b-5"><b>Lien vers la vidéo </b></h5>
<p class="text-muted m-b-30 font-13">
    <a href="https://vimeo.com/zxzzxzx" target="_blank">https://vimeo.com/zxzxzx</a>
</p>
"""

soup = BeautifulSoup(html, "lxml")

# Find the class `text-muted m-b-30 font-13` under a `p` which contains an `a`
print(soup.select_one("p.text-muted.m-b-30.font-13:has(a) a")["href"])

Output:

https://vimeo.com/zxzzxzx
user_123497340
Related