scrape emojis with text in beautiful soup

user_2378622 0
Viewed 743

I am trying to scrape a page using python and beautiful soup bs4

I want to keep the text in the <p> element in the page along with the emojis in this text.

The first attempt was:

import urllib
import urllib.request
from bs4 import BeautifulSoup

urlobject = urllib.request.urlopen("https://example.com")

soup = BeautifulSoup(urlobject, "lxml")

result= list(map(lambda e: e.getText(), soup.find_all("p", {"class": "text"})))

But this doesn't include emojis. I then tried to remove .getText() and just keep :

result= list(map(lambda e: e, soup.find_all("p", {"class": "text"})))

Which made me realize the emojis in this website are in the alt of img tags:

<p class="text">I love the night<img alt="" class="emoji" src="etc"/><span>!</span></p>

So what I want to do is :

  • getText() for p with class text
  • But also get alt for img with class=emoji

And keep the text and the emojis as one sentence.

Is there any way to do this?

Any help would be appreciated.

emoji beautifulsoup python
2 Answers

How about the following, returning tuples of the targeted data for each p? I just used your example p element twice as the input for this test:

from bs4 import BeautifulSoup

s = """
<p class="text">I love the night<img alt="" class="emoji" src="etc"/><span>!</span></p>
<p class="text">I love the night<img alt="" class="emoji" src="etc"/><span>!</span></p>
"""

soup = BeautifulSoup(s, 'lxml')

elements = soup.find_all('p', {'class': 'text'})
print(list(map(lambda e: (e.getText(), e.find('img', {'class': 'emoji'})['alt']), elements)))

Result:

[('I love the night!', ''), ('I love the night!', '')]
user_61244110

if the img.emoji are optional you can try below, and it will preserve emoji position

urlobject = '''<p class="text">I love the night<img alt="" class="emoji" src="etc"/><span>!</span></p>
<p class="text">I love the day<span>!</span></p>
<p class="text">I love the music<img alt="" class="emoji" src="etc"/> <img alt="" class="emoji" src="etc"/><span>!</span></p>
'''

result = []
for p in soup.find_all('p', {'class': 'text'}):
    emoji = p.select('img.emoji')
    if emoji:
        for em in emoji:
            index = p.contents.index(em)
            p.contents[index].replace_with(em['alt'])
    result.append(p.getText())

print(result)

Results:

['I love the night!', 'I love the day!', 'I love the music !']
user_4582140
Related