href beautifulsoul html selection doesn't return wanted result

Viewed 51

I want to extract all the first href of the first link by beautiful soup as shown my code HTML

<body> 
    <div>
        some html
        <footer>
            <div>...</div>
            <div>
                <span></span>
                <span></span>
                <a href="some link alpha"></a>      #<<<<<<------- i want this
                <a href="some link beta"></a>
                </div>
            </footer>
        </div>
    
        <div>
            some html
            <footer>
                <div>...</div>
                <div>
                    <span></span>
                    <span></span>
                    <a href="somelink gamma"></a>  #<<<<<<------- i want this too
                    <a href="somelink delta"></a>
                </div>
            </footer>
        </div>
....
    </body> 

i want to return an array of the wanted links, in this case it's [alpha,gamma]

here is my python code:

html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')
content = soup.find_all('footer div a',attrs={'href' : True})

for a in content:
    print ("Found the URL:", a['href'])

the code is not giving what i want, if i make only 'a' it returns all the links, i tried many things but no given results, even css selector didn't work for me

1 Answers

Try this:

from bs4 import BeautifulSoup

sample_html = """
    <div>
        some html
        <footer>
            <div>...</div>
            <div>
                <span></span>
                <span></span>
                <a href="link1 i want"></a>      #<<<<<<------- i want this
                <a href="unwanted link1"></a>
                </div>
            </footer>
        </div>
    
        <div>
            some html
            <footer>
                <div>...</div>
                <div>
                    <span></span>
                    <span></span>
                    <a href="link2 i want"></a>  #<<<<<<------- i want this too
                    <a href="unwanted link2"></a>
                </div>
            </footer>
        </div>
"""

links_you_want = [
    footer.find("a")["href"] for footer in 
    BeautifulSoup(sample_html, "html.parser").find_all("footer")
]

print(links_you_want)

Output:

 ['link1 i want', 'link2 i want']
Related