Array length Error while creating Dataframe from Scraped Data using BeautifulSoup

user_13892646 0
Viewed 77

I am creating a dataset of IMDB Ratings and Reviews.
Link
I want to scrape all the ratings and reviews on this page. There are certain reviews without ratings, because of which my count of reviews and ratings is different.
I have tried various ways to handle null values but was not able to implement them successfully.

My Code:

import requests
from bs4 import BeautifulSoup
import re
import pandas as pd
import numpy as np
import itertools
import string

url = (
    "https://www.imdb.com/title/tt6320628/reviews/_ajax?ref_=undefined&paginationKey={}"
)
key = ""
data = {"user_id": [], "rating":[], "title": [], "review": []}

while True:
    response = requests.get(url.format(key))
    soup = BeautifulSoup(response.content, "html.parser")
    # Find the pagination key
    pagination_key = soup.find("div", class_="load-more-data")
    if not pagination_key:
        break

    for user in (
        [tag.attrs['href'] for tag in soup.find_all('a', attrs={'class': None})
                if tag.attrs['href'].startswith('/user') & tag.attrs['href'].endswith('/')]
    ):
        data["user_id"].append(user[6:-1])

    for rate in (
        [tag.previous_element for tag in soup.find_all('span', attrs={'class': 'point-scale'})]
    ):
      if (rate.__eq__(None)):
        data["rating"].append(None)
      else:
        data["rating"].append(rate)
    
    ## Update the 'key' variable in-order to scrape more reviews
    key = pagination_key["data-key"]
    for title, review in zip(
        soup.find_all(class_="title"), soup.find_all(class_="text show-more__control")
    ):
        data["title"].append(title.get_text(strip=True))
        data["review"].append(review.get_text())
  

df = pd.DataFrame(data)
print(df)

len(data['rating'])
>>>2107

len(data['review'])
>>>2150

Error:

ValueError                                Traceback (most recent call last)
<ipython-input-28-0064f972ba2a> in <module>()
     41 
     42 
---> 43 df = pd.DataFrame(data)
     44 print(df)

3 frames
/usr/local/lib/python3.7/dist-packages/pandas/core/internals/construction.py in extract_index(data)
    395             lengths = list(set(raw_lengths))
    396             if len(lengths) > 1:
--> 397                 raise ValueError("arrays must all be same length")
    398 
    399             if have_dicts:

ValueError: arrays must all be same length

I want to have blank values for the ratings not available in the dataframe.

nonetype dataframe beautifulsoup python
1 Answers

Unfortunately there isn't always a rating so the logic here fails:

for rate in (
        [tag.previous_element for tag in soup.find_all('span', attrs={'class': 'point-scale'})]
    ):
      if (rate.__eq__(None)):
        data["rating"].append(None)
      else:
        data["rating"].append(rate)

You end up looping over less than the expected number of items irrespective of what you append.

One possible solution:

You need to amend to ensure you loop the same number of items as for other lists e.g.

for rate in (
    [tag.select_one('.point-scale').previous_element if tag.select_one('.point-scale') is not None else None 
     for tag in soup.select('.lister-item-content')] 
):
    data["rating"].append(rate)

Side note:

You could debug by adding in, below:

if not pagination_key:
    break

the following:

if len(soup.select('.lister-item-content, .point-scale')) % 2:
    print(url.format(key))
    break

then visit that printed url in the browser and enter .lister-item-content, .point-scale in the elements tab browser find box and hit return; if you get an uneven number of matches, then there is a missing rating and you can cycle through the reviews to see where.

user_62412350
Related