Unable to convert merged Dask dataframe to pandas dataframe - Getting the error "Unalignable boolean Series provided as indexer"

user_14795943 0
Viewed 105

I am trying to merge 3 Dask dataframes and query them accordingly. The dataframes are created from .csv files of SQL tables of formula1 data. The tables

laptimes: raceId, driverId, lap, position, time
races: raceId, year, round
drivers: driverId, forename, surname

The laptimes table below has foreign-key relationships with a races and drivers table. I have read in, merged the dataframes and attempted to query the dataframe:

from dask import dataframe as df
import pandas as pd
pd.set_option("max_rows", None)

df_times = df.read_csv("lap_times.csv", blocksize=64000000)
df_races = df.read_csv("races.csv",blocksize=64000000)
df_drivers = df.read_csv("drivers.csv", blocksize=64000000)

pd1 = df_times.merge(df_drivers, on = "driverId").merge(df_races, on = "raceId")\
        .drop(labels = ["dob", "nationality", "milliseconds", "url_x", "url_y", "time_y", "date"], 
                axis = 1)\
                .loc[(df_drivers["forename"] == "Sebastian") 
                    & (df_drivers["surname"] == "Vettel") 
                    & (df_races["year"] == 2018) 
                    & (df_races["round"] == 1), :]

I am able to run the above code with no errors.However If I try to convert the filtered dask dataframe to a pandas dataframe, using pd1.compute(), I get the error:

Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match).

I have searched for solutions for this and seen that, with a pandas dataframe, resetting the index is a solution, but does not work with a dask dataframe. What am I missing? Any help would be appreciated.

dask-dataframe dask python
1 Answers

I was able to obtain the desired subsets, but I've had to perform each merge seperately:

pd1 = df_times.merge(df_drivers, on = "driverId")\
        .drop(labels = ["dob", "nationality", "milliseconds", "url"],axis = 1)\

pd1 = pd1.merge(df_races, on = "raceId")
        
pd2 = pd1.drop(["url", "time_y", "date"], axis = 1)

pd2.loc[(pd2["forename"] == "Lewis") & (pd2["surname"] == "Hamilton") & (pd2["year"] == 2018) & (pd2["round"] == 1)].compute()

This is not optimal and I would still like to find a solution, to performing the merge between all 3 tables in one statement.

user_147959430
Related