How do I deal with an html table with merged (colspan = 2) columns in Python (preferably with Beautifulsoup)?

user_10857227 0
Viewed 61

I'm trying to extract the preference distribution data from the second table on this page. For context, the plan is to identify the party of each candidate and see how many votes my preferred party had before they were eliminated from the count. This is my first attempt at web scraping, so at considerable personal distress, I have managed to parse the page and get the data from the relevant table.

from bs4 import BeautifulSoup

    # Open and read html
    f = open("https://results.ecq.qld.gov.au/elections/state/State2017/results/booth1.html", "r")
    contents = f.read()
    
    # Parse the html data and then get to the preference distribution table
    soup = BeautifulSoup(contents, 'html.parser')

    useful_data = (soup.find_all(class_="resultTableBorder")[2].find_all("tr")[1:])

    # Extract the results of the preference distribution
    data = []
    for row in list(useful_data): 
        sub_data = []    
        for cell in row.find_all("td"): 
            try: 
                #target = 
                sub_data.append(cell.get_text(strip = True))
                target = ""
            except: 
                continue
        data.append(sub_data)
        sub_data = []

However, when I check to see if I have a nicely formatted list of lists, I do not.

    # Check if I have a nicely formed table of data. I do not.
    for index, row in enumerate(data, start = 1):
        try:
            length = len(row)
            print("Row " + str(index) + " contains " + str(length) + " elements.")
        except:
            continue

This yields the below, indicating that it's going to be a fussy mess to match up the column headers with the relevant data, ignore the horizontal lines and deal with different numbers of candidates (there are 93 electorates, this is just the first).

Row 1 contains 8 elements.
Row 2 contains 10 elements.
Row 3 contains 1 elements.
Row 4 contains 13 elements.
Row 5 contains 13 elements.
Row 6 contains 13 elements.
Row 7 contains 13 elements.
Row 8 contains 1 elements.
Row 9 contains 13 elements.
Row 10 contains 1 elements.
Row 11 contains 5 elements.
Row 12 contains 2 elements.
Row 13 contains 2 elements.
Row 14 contains 1 elements.

Is there an easy way to do this, either using a clever trick when I'm extracting the preference distribution or a way of dealing with the data I extract?

html-table beautifulsoup html python
1 Answers

In this case, it's easier to do it this way:

import pandas as pd
tables = pd.read_html('https://results.ecq.qld.gov.au/elections/state/State2017/results/booth1.html')
target_df = tables[5] #this is the Summary of Distribution of Preferences table
target_df.drop(target_df.tail(3).index).iloc[1:].dropna(how='all') #a little clean up

This should get you your target table. If necessary, you can do more clean up, formatting etc. Or extract to lists using standard pandas methods.

user_94480900
Related