BeautifulSoup4 scraping: Pandas "arrays must all be same length" when exporting data to csv

user_7368045 0
Viewed 782

I'm using BeautifulSoup4 to scrape info from a website and using Pandas to export the data to a csv file. There are 5 columns of data represented by 5 lists in a dictionary. However, since the website doesn't have complete data for all 5 categories, some lists have fewer items than others. So when I try to export the data, pandas gives me

ValueError: arrays must all be same length.

What is the best way to handle this situation? To be specific, the lists with fewer items are "authors" and "pages". Thanks in advance!
Code:

import requests as r
from bs4 import BeautifulSoup as soup
import pandas 

#make a list of all web pages' urls
webpages=[]
for i in range(15):
    root_url = 'https://cross-currents.berkeley.edu/archives?author=&title=&type=All&issue=All&region=All&page='+ str(i)
    webpages.append(root_url)
print(webpages)
#start looping through all pages
titles = []
journals = []
authors = []
pages = []
dates = []
issues = []

for item in webpages:
    headers = {'User-Agent': 'Mozilla/5.0'}
    data = r.get(item, headers=headers)
    page_soup = soup(data.text, 'html.parser')

    #find targeted info and put them into a list to be exported to a csv file via pandas
    title_list = [title.text for title in page_soup.find_all('div', {'class':'field field-name-node-title'})]
    titles += [el.replace('\n', '') for el in title_list]

    journal_list = [journal.text for journal in page_soup.find_all('em')]
    journals += [el.replace('\n', '') for el in journal_list] 

    author_list = [author.text for author in page_soup.find_all('div', {'class':'field field--name-field-citation-authors field--type-string field--label-hidden field__item'})]
    authors += [el.replace('\n', '') for el in author_list]

    pages_list = [pages.text for pages in page_soup.find_all('div', {'class':'field field--name-field-citation-pages field--type-string field--label-hidden field__item'})]
    pages += [el.replace('\n', '') for el in pages_list]

    date_list = [date.text for date in page_soup.find_all('div', {'class':'field field--name-field-date field--type-datetime field--label-hidden field__item'})]
    dates += [el.replace('\n', '') for el in date_list]

    issue_list = [issue.text for issue in page_soup.find_all('div', {'class':'field field--name-field-issue-number field--type-integer field--label-hidden field__item'})]
    issues += [el.replace('\n', '') for el in issue_list]

# export to csv file via pandas
dataset = {'Title': titles, 'Author': authors, 'Journal': journals, 'Date': dates, 'Issue': issues, 'Pages': pages}
df = pandas.DataFrame(dataset)
df.index.name = 'ArticleID'
df.to_csv('example45.csv', encoding="utf-8")
python-3.6 pandas beautifulsoup python
2 Answers

If you are sure, that for examples the length of the titles is always correct, you could do something like that:

title_list = [title.text for title in page_soup.find_all('div', {'class':'field field-name-node-title'})]
titles_to_add = [el.replace('\n', '') for el in title_list]
titles += titles_to_add

...

author_list = [author.text for author in page_soup.find_all('div', {'class':'field field--name-field-citation-authors field--type-string field--label-hidden field__item'})]
authors_to_add = [el.replace('\n', '') for el in author_list]
if len(authors_to_add) < len(titles_to_add):
    while len(authors_to_add) < len(titles_to_add):
        authors_to_add += " "    
authors += authors_to_add

pages_list = [pages.text for pages in page_soup.find_all('div', {'class':'field field--name-field-citation-pages field--type-string field--label-hidden field__item'})]
pages_to_add = [el.replace('\n', '') for el in pages_list]
if len(pages_to_add) < len(titles_to_add):
    while len(pages_to_add) < len(titles_to_add):
        pages_to_add += " "
pages += pages_to_add

However... This will just add elements to the columns, so that they have the correct length so that you can create a dataframe. But in your dataframe the Authors and pages will not be in the correct row. You will have to change your algorithm a bit to achieve your final goal... It would be better if you would iterate through all rows on your page and get title etc... like this:

rows = page_soup.find_all('div', {'class':'views-row'})
    for row in rows:
        title_list = [title.text for title in row.find_all('div', {'class':'field field-name-node-title'})]
...

Then you need to check if a title, author, etc... exists len(title_list)>0and if not, add "None" or something else to the specific list. Then everything should be correct in your df.

user_84155410

You could make a dataframe out of just the first list (df = pandas.DataFrame({'Title': titles})), and then add the others:

dataset = {'Author': authors, 'Journal': journals, 'Date': dates, 'Issue': issues, 'Pages': pages}
df2 = pandas.DataFrame(dataset) 
df_final = pandas.concat([df, df2], axis=1)

This will give you blanks (or NaN) where you have missing data.

The trouble with this, as with @WurzelseppQX's answer, is that the data may not be aligned, which would make it pretty useless. So the best is perhaps to change your code in such a way that you always append something to the each list for each run through the loop, just making it 0 or blank if there's nothing there.

user_62207590
Related