Parse each row of Pandas df to extract list of actors from each URL

camhines4 0
Viewed 26

I am newly learning how to do web scraping in Python and could use some help. I want to parse each url in the below table and extract a list of the cast members in each movie.

Date Name Year Letterboxd URI Rating
2020-04-06 Knives Out 2019 https://boxd.it/jWEA 4.0
2020-04-07 Pulp Fiction 1994 https://boxd.it/29Pq 5.0

Ultimate goal is to have a new df that shows each cast member that gets extracted, matching them to the movie and rating as well.

Actor Movie Rating
Daniel Craig Knives Out 4.0
Bruce Willis Pulp Fiction 5.0
beautifulsoup web-scraping pandas python
1 Answers

This might be what you are looking for.

import pandas as pd
import requests
from bs4 import BeautifulSoup as bs

# replicate given data
columns = ['Date', 'Name', 'Year', 'Letterboxd URI', 'Rating']
r1 = ['2020-04-06', 'Knives Out', '2019', 'https://boxd.it/jWEA', '4.0']
r2 = ['2020-04-07', 'Pulp Fiction', '1994', 'https://boxd.it/29Pq', '5.0']
df_1 = pd.DataFrame([r1], columns=columns)
df_2 = pd.DataFrame([r2], columns=columns)
df = pd.concat([df_1, df_2])


columns = ['Movie', 'Actor', 'Rating']
new_df = pd.DataFrame(columns=columns)

for index, row in df.iterrows():
    
    # request content and fetching a-tag inside cast_list div
    r = requests.get(row['Letterboxd URI'])
    soup = bs(r.content, 'lxml')
    # as each cast encapsulate inside <a> and <div cast-list...>
    cast_soup = soup.select_one('[class="cast-list text-sluglist"]')
    cast_list = [a.get_text().strip() for a in cast_soup.find_all("a")]
    
    # putting data into new_df
    movie, rating = row['Name'], row['Rating'] # old value from tables
    for cast in cast_list:
        df_ = pd.DataFrame([[movie, cast, rating]], columns=columns)
        new_df = pd.concat([new_df, df_])
        
new_df = new_df.reset_index(drop=True)
new_df

|    | Movie        | Actor                   |   Rating |
|---:|:-------------|:------------------------|---------:|
|  0 | Knives Out   | Daniel Craig            |        4 |
|  1 | Knives Out   | Chris Evans             |        4 |
|  2 | Knives Out   | Ana de Armas            |        4 |
|  3 | Knives Out   | Jamie Lee Curtis        |        4 |
| 33 | Pulp Fiction | John Travolta           |        5 |
| 34 | Pulp Fiction | Samuel L. Jackson       |        5 |
| 35 | Pulp Fiction | Uma Thurman             |        5 |
brixk0
Related