I wish to receive only links that match a specific criteria

user_15809519 0
Viewed 42

I have this Selenium code:

driver.get("https://www.youtube.com/")

link_mix = driver.find_elements_by_tag_name('a')

for linked in link_mix:
    print(linked.get_attribute('href'))

I wish to get links that are of the form youtube.com/contentcodexxxxxx and not any other hyperlinks such as header links.

web selenium screen-scraping python
2 Answers

You can approach this from many angles.

First one that is pretty straightforward is to just check for that string in the URL.

link_mix = [x for x in link_mix if "youtube.com/contentcode" in x.get_attribute('href')]

Additionally, instead of doing

link_mix = driver.find_elements_by_tag_name('a')

You could narrow it down by specifying an html element where the links you are looking for are located.

html_body = driver.find_element_by_tag_name('body')
link_mix = html_body.find_elements_by_tag_name('a')

This will only find hyperlinks inside the body of the html, ignoring links in header or footer if the webpages contains them. If you know that these links can be narrowed down even further in another html element, you can replace the first "find_element". Do note that find_element_by_tag_name returns an exception if no such tag is found.

user_157973110

Selenium does not provide any way of finding link tags with any filters.

Instead you must iterate through the elements you have retrieved and discard any that you do not want.

driver.get("https://www.youtube.com/")

link_mix = driver.find_elements_by_tag_name('a')

for linked in link_mix:
    if not linked.get_attribute('href').startswith('youtube.com/contentcode'):
        continue
    print(linked.get_attribute('href'))
user_13574090
Related