Extracting href from 'a' element with text only attribute

user_14198522 0
Viewed 98

I am trying to build a function in a python webscraper that moves to the next page in a list of results. I am having trouble locating the element in beautiful soup as the link is found at the end of many other tags, and doesn't have any attributes such as class or ID.

Here is a snippet of the html:

<a href="http://www.url?&=page=2">
     Next
    
   </a>

I have been reading the bs4 documentation trying to understand how I can extract the URL, but I am coming up stumped. I am thinking that it could be done by either:

  1. finding the last .a['href'] in the parent element, as it is always the last one.
  2. finding the href based on the fact that it always has text of 'Next'

I don't know how to write something that would solve either 1. or 2.

Am I along the right lines? Does anyone have any suggestions to achieve my goal? Thanks

beautifulsoup python
1 Answers

To find <a> tag that contains text Next, you can do:

from bs4 import BeautifulSoup


txt = '''
<a href="http://www.url?&=page=2">
     Next
    
   </a>'''


soup = BeautifulSoup(txt, 'html.parser')    
print(soup.select_one('a:contains("Next")')['href'])

Prints:

http://www.url?&=page=2

Or:

print(soup.find('a', text=lambda t: t.strip() == 'Next')['href'])

To get the last <a> tag inside some element, you can index the ResultSet with [-1]:

from bs4 import BeautifulSoup


txt = '''
<div id="block">
    <a href="#">Some other link</a>
    <a href="http://www.url?&=page=2">Next</a>
</div>
'''


soup = BeautifulSoup(txt, 'html.parser')

print(soup.select('div#block > a')[-1]['href'])
user_100359850
Related