Web scraping using Beautiful Soup, scraping multiple elements without class

user_8062990 0
Viewed 123

So I want to scrape the Director from this. But as I have seen the page I know there are two directors for this particular movie Danny Boyle and Loveleen Tandan. But there is no way to get it if I use find_all('a') then It will also take the name of actors like Dev Patel, Freida Pinto.

I can't use find_all('a')[1] and find_all('a')[2] because for other movies there could be only one directors. The only thing seperating actors from director is span tag with class ghost. How should I scrape this data, assuming there could be one, two or three directors.

<p class="">
             Directors:
             <a href="/name/nm0000965/">
              Danny Boyle
             </a>
             ,
             <a href="/name/nm0849164/">
              Loveleen Tandan
             </a>
             <span class="ghost">
              |
             </span>
             Stars:
             <a href="/name/nm2353862/">
              Dev Patel
             </a>
             ,
             <a href="/name/nm2951768/">
              Freida Pinto
             </a>
             ,
             <a href="/name/nm0795661/">
              Saurabh Shukla
             </a>
             ,
             <a href="/name/nm0438463/">
              Anil Kapoor
             </a>
            </p>

The url of the page is : https://www.imdb.com/search/title/?count=100&groups=oscar_best_picture_winners&sort=year%2Cdesc&ref_=nv_ch_osc

web-scraping beautifulsoup python
2 Answers

This should help u:

from bs4 import BeautifulSoup

html = """
<p class="">
             Directors:
             <a href="/name/nm0000965/">
              Danny Boyle
             </a>
             ,
             <a href="/name/nm0849164/">
              Loveleen Tandan
             </a>
             <span class="ghost">
              |
             </span>
             Stars:
             <a href="/name/nm2353862/">
              Dev Patel
             </a>
             ,
             <a href="/name/nm2951768/">
              Freida Pinto
             </a>
             ,
             <a href="/name/nm0795661/">
              Saurabh Shukla
             </a>
             ,
             <a href="/name/nm0438463/">
              Anil Kapoor
             </a>
            </p>
""" #The html code provided by you

soup = BeautifulSoup(html,'html5lib')

p_tag = soup.find('p')

span = p_tag.find('span',class_ = "ghost")

prev = list(span.previous_siblings) #Finds all the tags before the span tag with class ghost and converts them into a list

prev = [str(x) for x in prev]

prev = ''.join(prev) #Converts the list to a string

soup2 = BeautifulSoup(prev,'html5lib') #Creates a new BeautifulSoup object with the newly formed string

a_tags = soup2.find_all('a')

for a in a_tags:
    txt = a.text.strip()
    print(txt)

Output:

Loveleen Tandan
Danny Boyle

Hope that this helps!

user_123098080

I just leave it here:

import requests
from bs4 import BeautifulSoup


url = 'https://www.imdb.com/search/title/?count=100&groups=oscar_best_picture_winners&sort=year%2Cdesc&ref_=nv_ch_osc'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

directors_and_stars = soup.find_all(text=lambda t: 'Director' in t)

for d in directors_and_stars:
    movie_name = d.find_previous('h3').a.get_text(strip=True)
    directors = [t.strip() for t in d.find_previous('p').find_all(text=True)[1:] if t.strip() and t.strip() != ',']
    directors = directors[:directors.index('|')]
    print('{:<50} {}'.format(movie_name, directors))

Prints:

Parazit                                            ['Bong Joon Ho']
Zelená kniha                                       ['Peter Farrelly']
The Shape of Water                                 ['Guillermo del Toro']
Moonlight                                          ['Barry Jenkins']
Spotlight                                          ['Tom McCarthy']
Birdman or (The Unexpected Virtue of Ignorance)    ['Alejandro G. Iñárritu']
12 rokov otrokom                                   ['Steve McQueen']
Argo                                               ['Ben Affleck']
The Artist                                         ['Michel Hazanavicius']
Králova rec                                        ['Tom Hooper']
Slumdog Millionaire                                ['Danny Boyle', 'Loveleen Tandan']
Smrt' caká vsade                                   ['Kathryn Bigelow']
Táto krajina nie je pre starých                    ['Ethan Coen', 'Joel Coen']

...and so on.
user_100359850
Related