How to filter certain URLs from a list based on keywords

Kelly Tang 0
Viewed 47

I am having problem with removing certain URLs from a list in Python. I want to know the easiest and quickest way.

I got a list of URLs returned from google search API. I want to remove all the websites which have the domain "trip advisor", "facebook", "instagram", "twitter" and "ebag".

Here is what I have tried so far:

page_urls = ['https://the1955club.com/', 'https://www.tripadvisor.com/Restaurant_Review-g1842228-d10140470-Reviews-The_1955_Club-Walton_On_Thames_Surrey_England.html']
# print('page_urls', page_urls)

all_urls = []

for address in page_urls:
    url = urlparse(address)
    new_url = url.netloc
    all_urls.append(new_url)

all_urls.remove('www.tripadvisor.com')
all_urls.remove('tripadvisor.com')

I am getting this error:

ValueError: list.remove(x): x not in list
list python
1 Answers

You can do:

from urllib.parse import urlparse

page_urls = [
    "https://the1955club.com/",
    "https://www.tripadvisor.com/Restaurant_Review-g1842228-d10140470-Reviews-The_1955_Club-Walton_On_Thames_Surrey_England.html",
]

forbiddens = ["facebook", "instagram", "twitter", "tripadvisor"]


def check_url(url):
    parsed_url = urlparse(url).netloc
    return not any(item in parsed_url for item in forbiddens)


valid_urls = [url for url in page_urls if check_url(url)]
print(valid_urls)

Basically you first parse the URL with urllib.parse.urlparse and get the netloc part of it. Next you iterate through your forbidden names and check to see if any of them is in the netloc. check_url does the filtering.

This of course doesn't remove URLs from page_urls, it creates new one. But you can do this if you want to mutate that list:

page_urls[:] = [url for url in page_urls if check_url(url)]
print(page_urls)
S.B0
Related