BeautifulSoup - search text inside a tag

user_17054116 0
Viewed 43
from bs4 import BeautifulSoup
from fake_useragent import UserAgent
import requests
user = UserAgent()
headers = {
    'user-agent' : user.random
}
url = 'https://www.wildberries.ru/?utm_source=domain&utm_campaign=wilberes.ru'
def main():
    resp = requests.get(url, headers=headers)
    soup = BeautifulSoup(resp.text, 'lxml')
    main = soup.find('div', class_='menu-burger__main')
    ul = main.find('ul', class_='menu-burger__main-list')
    all = ul.find_all_next('li', class_='menu-burger__main-list-item')
    f = open('link.txt', 'a')
    for lin in all:
        get_link = lin.find('a').get('href')
        f.write(get_link + '\n')
    f.close()
if __name__ == '__main__':
    main()

I'm trying to parse a link to a section and its name. I managed to get the link, but how can I get the name if it is not in the tag?

screenshot of a browser with the Dec tools open highlighting an HTML element

beautifulsoup html python
1 Answers

Using the string property seems to be the correct method based on BS4 documentation:

myLink = soup.find('a')
myLinkText = str(myLink.string)

The idea of using str() is to convert the text to regular python strings, as .string returns a BS4 NavigableString object (you may find you don't really have to do this, though - it might be safer though, as well as stripping whitespace from the string so that you don't get weird newlines or padding with the result -- str(myLink.string).strip())

Reference: https://crummy.com/software/BeautifulSoup/bs4/doc/#navigablestring

In your code, you are actually getting the href, so take note that in my code above I am getting the anchor tag, not just it's href attribute.

user_14679140
Related