Getting incorrect link in BeautifulSoup python web scraper

user_18871432 0
Viewed 74

I am writing a web scraper and am struggling to get the href link from a web page. The URL is https://www.seedinvest.com/auto I am trying to get the href links of their individual articles. Here is an example:

<a class="card-url full-size-content" href="https://www.seedinvest.com/soil.connect/seed"></a>

Here is my code:

import pandas as pd
import os
import requests
from bs4 import BeautifulSoup
import re



URL = "https://www.seedinvest.com/offerings"
page = requests.get(URL)
soup = BeautifulSoup(page.text, "html.parser")


### Searching for all embeded company website links and displaying them

links = []
for link in soup.findAll(class_="card-url full-size-content"):
    links.append(link.get('href'))
print(links)

When I run my code, I get this:

[]

Can you help me find the right links?

beautifulsoup python
2 Answers

It's working in url

import pandas as pd
import os
import requests
from bs4 import BeautifulSoup
import re


URL = "https://www.seedinvest.com/auto"
page = requests.get(URL)
soup = BeautifulSoup(page.text, "html.parser")


### Searching for all embeded company website links and displaying them

links = []
for link in soup.findAll(class_="card-url full-size-content"):
    links.append(link.get('href'))
print(links)

Output:

['https://www.seedinvest.com/nowrx/series.c', 'https://www.seedinvest.com/appmail/seed', 'https://www.seedinvest.com/soil.connect/seed', 'https://www.seedinvest.com/cytonics/series.c.2']
user_128484110

Maybe you're using the wrong URL in your code: https://www.seedinvest.com/offerings instead of https://www.seedinvest.com/auto?

user_187351130
Related