How to extract url/links that are contents of a webpage with BeautifulSoup

user_10009287 0
Viewed 386

So the website I am using is : https://keithgalli.github.io/web-scraping/webpage.html and I want to extract all the social media links on the webpage.

import requests 
from bs4 import BeautifulSoup as bs

r = requests.get('https://keithgalli.github.io/web-scraping/webpage.html')
soup = bs(r.content)

links = soup.find_all('a', {'class':'socials'})

actual_links = [link['href'] for link in links]

I get an error, specifically:

KeyError: 'href'

For a different example and webpage, I was able to use the same code to extract the webpage link but for some reason this time it is not working and I don't know why.

I also tried to see what the problem was specifically and it appears that links is a nested array where links[0] outputs the entire content of the ul tag that has class=socials so its not iterable so to speak since the first element contains all the links rather than having each social li tag be seperate elements inside links

web-scraping beautifulsoup python
2 Answers

Why not try something like:

import requests 
from bs4 import BeautifulSoup as bs

r = requests.get('https://keithgalli.github.io/web- 
scraping/webpage.html')
soup = bs(r.content)

links = soup.find_all('a', {'class':'socials'})

actual_links = [link['href'] for link in links if 'href' in link.keys()]

After gaining some new information from you and visiting the webpage, I've realized that you did the following mistake:

enter image description here

The socials class is never used in any a-element and thus you won't find any such in your script. Instead you should look for the li-elements with the class "social".

Thus your code should look like:

import requests 
from bs4 import BeautifulSoup as bs

r = requests.get('https://keithgalli.github.io/web- 
scraping/webpage.html')
soup = bs(r.content, "lxml")

link_list_items = soup.find_all('li', {'class':'social'})
links = [item.find('a').get('href') for item in link_list_items]

print(links)
user_128607160

Here is the solution using css selectors:

import requests 
from bs4 import BeautifulSoup as bs

r = requests.get('https://keithgalli.github.io/web-scraping/webpage.html')
soup = bs(r.content, 'lxml')

links = soup.select('ul.socials li a')

actual_links = [link['href'] for link in links]

print(actual_links)

Output:

['https://www.instagram.com/keithgalli/', 'https://twitter.com/keithgalli', 'https://www.linkedin.com/in/keithgalli/', 'https://www.tiktok.com/@keithgalli']
user_128484110
Related