Scraping URLs in Beautiful Soup but not getting all links

user_17099936 0
Viewed 44

This is the relevant part of the html, that is being scraped:

<div class="blockSpoiler-content">
   <div class="contentSpoiler">
      <div class="link-box" id="62H" style="background-color: rgb(65, 120, 50);">
         <div class="status-box"><i class="working" title="Working"></i></div>
         <a rel="external" href="https://url1.net.html" target="_blank">Link1</a>
      </div>
      <div class="link-box" id="IFA" style="background-color: rgb(65, 120, 50);">
         <div class="status-box"><i class="working" title="Working"></i></div>
         <a rel="external" href="https://url2.net.html" target="_blank">Link2</a>
      </div>
      <div class="link-box" id="ruG" style="background-color: rgb(65, 120, 50);">
         <div class="status-box"><i class="working" title="Working"></i></div>
         <a rel="external" href="https://url3.com.html" target="_blank">Link3</a>
      </div>
      <div class="link-box" id="Bdf" style="background-color: rgb(65, 120, 50);">
         <div class="status-box"><i class="working" title="Working"></i></div>
         <a rel="external" href="https://url4.com" target="_blank">Link4</a>
      </div>
      <div class="link-box" id="1Da" style="background-color: rgb(65, 120, 50);">
         <div class="status-box"><i class="working" title="Working"></i></div>
         <a rel="external" href="https://url5.net.html" target="_blank">Link5</a>
      </div>
   </div>
</div>

I am trying to get these URLs:

  1. https://url1.net.html
  2. https://url2.net.html
  3. https://url3.com.html
  4. https://url4.com
  5. https://url5.net.html

I tried different things, but only got till here (local file being used only for the purpose of testing, before web scraping):

with open("mainLocalFile.html") as fp:
soup2 = BeautifulSoup(fp, 'html.parser')
links = soup2.find_all('div', class_='blockSpoiler-content')
# print(links)
for link in links:
    print(link)
    print(link.a)          # prints only the first tag
    print(link.a['href'])  # prints only the first URL
web-scraping beautifulsoup python
1 Answers

Select all <a> which are under tag with class blockSpoiler-content (now you're only selecting one <div class=blockSpoiler-content> with the .find_all method):

for a in soup.select(".blockSpoiler-content a"):
    print(a["href"])

Prints:

https://url1.net.html
https://url2.net.html
https://url3.com.html
https://url4.com
https://url5.net.html
user_100359850
Related