Get a list of links with specific pattern from list of many links

user_14509344 0
Viewed 50

I am using BeautifulSoup and requests to scrape and parse some data from a website.

I have gotten to a point where I have an output list of links from the website I am scraping, but I need to narrow this list down to only the links I need.

Here is a photo of the output for my code: enter image description here

I only need the links with the pattern /cpc/datasheet/#####

I have tried re.findall(), re.search(), filter(), and a few other options with no luck.

python-requests web-scraping beautifulsoup parsing python
1 Answers

You can use the startswith() method check if the link startswith /cpc/datasheet/ and append it to a list:

links = [link for link in soup.find_all('a') if str(link.get('href')).startswith('/cpc/datasheet/')]

Or:

out = []
for tag in soup.find_all('a'):
    link = tag.get('href')
    if str(link).startswith('/cpc/datasheet/'):
        print(link)
        out.append(link)
user_123497340
Related