Can't arrange and print some fields from a webpage in some customized way

user_10568531 0
Viewed 48

I've created a script to parse the movie name, all cast, Produced by and Casting By from this webpage. I can parse the aforesaid fields from that page. However, what I can't do is arrange and print the items in some customized manner when the four items come into consideration. The script that I've written so far can print the items exactly the way I want when I include movie name and cast only. I wish to include Produced by and Casting By as well that you see in this image.

I've tried so far:

import requests
from bs4 import BeautifulSoup

link = 'https://www.imdb.com/title/tt0068646/fullcredits?ref_=tt_cl_sm#cast'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36'
    r = s.get(link)
    soup = BeautifulSoup(r.text,"lxml")
    movie_name = soup.select_one("h3[itemprop='name'] > a").get_text(strip=True)
    for item in soup.select("h4#cast + table.cast_list tr:has(:not(.castlist_label))"):
        performer = item.select_one("td:not(.primary_photo) > a[href^='/name/']").get_text(strip=True)
        character = ' '.join(item.select_one("td.character").text.split())
        print(movie_name,performer,character)

Output I'm getting (movie name and cast):

The Godfather Marlon Brando Don Vito Corleone
The Godfather Al Pacino Michael Corleone
The Godfather James Caan Sonny Corleone
The Godfather Richard S. Castellano Clemenza (as Richard Castellano)
The Godfather Robert Duvall Tom Hagen
The Godfather Sterling Hayden Capt. McCluskey
The Godfather John Marley Jack Woltz
and so on----------------------

I wish to add the following results (taken from the two fields Produced by and Casting By that you see in the image) at the bottom of the above printing:

The Godfather Gray Frederickson associate producer
The Godfather Al Ruddy producer (as Albert S. Ruddy) (produced by)
The Godfather Robert Evans studio executive (uncredited)
The Godfather Louis DiGiaimo (casting)
The Godfather Andrea Eastman (casting)
The Godfather Fred Roos (casting)

How can I let the script print the fields the way I've shown above?

web-scraping beautifulsoup python-3.x python
1 Answers
import requests
from bs4 import BeautifulSoup

link = 'https://www.imdb.com/title/tt0068646/fullcredits?ref_=tt_cl_sm#cast'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36'
    r = s.get(link)
    soup = BeautifulSoup(r.text,"lxml")
    movie_name = soup.select_one("h3[itemprop='name'] > a").get_text(strip=True)
    for item in soup.select("h4#cast + table.cast_list tr:has(:not(.castlist_label))"):
        performer = item.select_one("td:not(.primary_photo) > a[href^='/name/']").get_text(strip=True)
        character = ' '.join(item.select_one("td.character").text.split())
        print(movie_name,performer,character)
    for row in soup.select('h4:contains("Produced by") + table tr'):
        name = row.select_one('.name').get_text(strip=True)
        credit = row.select_one('.credit').get_text(strip=True)
        print(movie_name, name, credit)
    for row in soup.select('h4:contains("Casting By") + table tr'):
        name = row.select_one('.name').get_text(strip=True)
        credit = row.select_one('.credit').get_text(strip=True)
        print(movie_name, name, credit)

Prints:

...
Krstný Otec Matthew Vlahakis Clemenza's Son (uncredited)
Krstný Otec Conrad Yama Fruit Vendor (uncredited)
Krstný Otec Gray Frederickson associate producer
Krstný Otec Al Ruddy producer (as Albert S. Ruddy) (produced by)
Krstný Otec Robert Evans studio executive (uncredited)
Krstný Otec Louis DiGiaimo (casting)
Krstný Otec Andrea Eastman (casting)
Krstný Otec Fred Roos (casting)

NOTE: Krstný Otec means Godfather in Slovak (I got slovak version of HTML because my country's IP).

user_100359850
Related