How to retrieve text data from multiple HTML tags?

user_10543310 0
Viewed 229

I have the following HTML snippet output stored in variable named content of type bs4.element.Tag.

<li class="item">
                                            Alpha-tocopherol
                                            <em>see</em>
<a href="https://medlineplus.gov/vitamine.html">Vitamin E</a>
</li>

str(content) output:

'<li class="item">\n                                            Alpha-tocopherol\n                                            <em>see</em>\n<a href="https://medlineplus.gov/vitamine.html">Vitamin E</a>\n</li>'

I would like to use Python to get as output: ['Alpha-tocopherol', 'Vitamin E']. I tried the following but it is wrong:

regex = re.compile('(\w+\s+)\n')
regex.sub('', content.text).split()
web-scraping beautifulsoup python-3.x xpath regex
2 Answers

You can get the first tag by using the .contents method and then use the .find_next() method to search for an a tag.

from bs4 import BeautifulSoup

html = """
<li class="item">
Alpha-tocopherol
<em>see</em>
<a href="https://medlineplus.gov/vitamine.html">Vitamin E</a>
</li>
"""
soup = BeautifulSoup(html, "html.parser")

for tag in soup.find_all("li", class_="item"):
    print([tag.contents[0].strip(), tag.find_next("a").text])

Output:

['Alpha-tocopherol', 'Vitamin E']
user_123497340

Both of these will produce the list you want. With the first one though, it depends on how the html element is parsed. If there's page breaks \n, you'll have to do some extra parsing.

html = '''<li class="item">Alpha-tocopherol<em>see</em><a href="https://medlineplus.gov/vitamine.html">Vitamin E</a></li>'''
soup = BeautifulSoup(html, "html.parser")

soup.text.split('see') # option 1, get all text and parse accordingly from soup object

soup.find('li', class_='item').text.split('see') # option 2, get text from li element (seems like it'd be less efficient to do this)

Output

['Alpha-tocopherol, 'Vitamin E']
user_121334340
Related