How to extract text inside span without class name with BeautifulSoup in python

user_3200615 0
Viewed 3758

I am making a dictionary that looks up the word in Spanish and gives me its corresponding English translation. I am quite new to BeautifulSoup, so I am taking advantage of this opportunity to learn web scraping.

The webpage link is https://www.spanishdict.com/translate/rojo. I just typed 'rojo' for an example.

However, after I successfully extracted the translation, I would like to extract some example sentences as well, but here I encounter one problem. I cannot extract one of the sentences inside the <span> WITHOUT class name.

I tried

soup.find_all(name='div', class_='indent--FyTYr')

But many unnecessary information within that class popped up. I also noticed one answer from the post the link mentioned previous_sibling, but it didn't work.

The sample html code will look like:

<div class="indent--FyTYr">
  <div>
    <span>The sky turned red at sundown.</span>
    <span class="dash--SIa20"></span>
    <em class="exampleDesktop--3n1hN">El cielo se tornó rojo al atardecer.</em> 
  </div>
...
</div>

I would like to extract the sentence from the above html sample code. But I haven't found any useful method to locate and extract.

The sky turned red at sundown.

web-scraping beautifulsoup html python
3 Answers

If you're sure that the span will be first, you can simply use .find("span"): 

>>> soup.find_all(name='div', class_='indent--FyTYr')[0].find("span").text
'The sky turned red at sundown.'
user_63428120

You can iterate over the outermost div of id='dictionary-neodict-es' with a recursive function to account for the fact that there are multiple nested divs with class of indent--FyTYr:

from bs4 import BeautifulSoup as soup
import requests, bs4
def has_class(d, c):
   return any(c in i.attrs.get('class', []) or has_class(getattr(i, 'contents', []), c) for i in d if i != '\n' and not isinstance(i, bs4.NavigableString))

def get_sentences(d):
   if 'indent--FyTYr' in d.attrs.get('class', []) and not has_class(d.contents, 'indent--FyTYr'):
      yield [d.div.span.text, d.div.em.text]
   else:
      for i in filter(lambda x:x != '\n' and not isinstance(x, bs4.NavigableString), getattr(d, 'contents', [])):
         yield from get_sentences(i)


result = list(get_sentences(soup(requests.get('https://www.spanishdict.com/translate/rojo').text, 'html.parser').find('div', {'id':'dictionary-neodict-es'})))

Now, you have access to all sentences:

[['The sky turned red at sundown.', 'El cielo se tornó rojo al atardecer.'], ['No quiero ver esa propaganda roja.', "I don't want to see that red propaganda."], ['Ella cree que me veo mejor vestida de rojo, pero no estoy segura.', "She thinks I look best dressed in red, but I'm not sure."], ['Durante la Guerra Fría, a los izquierdistas se les llamaba rojos.', 'During the Cold War, the leftists were called reds.']]

To access your desired string:

print(result[0][0])

Output:

'The sky turned red at sundown.'
user_73267380

You could provide a list of css selectors and generate a list of matched elements then a dictionary to lookup translations. Swop the ordering so English phrases are keys and spanish are values. Requires bs4 4.7.1+. The matched items are all the ones where you have phrase - translation.

from bs4 import BeautifulSoup as bs
import requests

r = requests.get('https://www.spanishdict.com/translate/rojo')
soup = bs(r.content, 'lxml')
elems = [i.text for i in soup.select('span:has(+[class^="dash--"]), div:has(span:has(+[class^="dash--"])) em:not([title]), .ex, .ex_unit .tran_group, .idm, .idm_unit .tran_group, .cpd, .cpd_group .tran_group')]
results = {v if k!=elems[0] else k:k if k!=elems[0] else v for k,v in zip(elems[0::2],elems[1::2])} #reverse first item
print(results)

Otherwise, you can also list slice pairs out.

user_62412350
Related