Concatenate Excel Files using Dask

Question

I have 20 Excel files and need to concatenate them using Dask (I have already done it using pandas, but it will grow in the future). I have used the following solution found here: Reading multiple Excel files with Dask

But throws me an error: cannot concatenate object of type ''; only Series and DataFrame objs are valid

What I am assuming is that it does not create a Dataframe, tried the following code:

df = pd.DataFrame()

files = glob.glob(r"D:\XX\XX\XX\XX\XXX\*.xlsx")

# note we are wrapping in delayed only the function, not the arguments
delayeds = [dask.delayed(pd.read_excel)(i, skiprows=0) for i in files]

# the line below launches actual computations
results = dask.compute(delayeds)

# after computation is over the results object will 
# contain a list of pandas dataframes

df = pd.concat(results, ignore_index=True)

The original solution did not include df=pd.DataFrame(). Where is the mistake?

Thank you!

user17743486 · Answer

Using the following solution: Build a dask dataframe from a list of dask delayed objects

Realized that the last line was not using dask but pandas. Changed the data to a numpy array to pandas.

Here is the code:

files = glob.glob(r"D:\XX\XX\XX\XX\XXX\*.xlsx")

# note we are wrapping in delayed only the function, not the arguments
delayeds = [dask.delayed(pd.read_excel)(i, skiprows=0) for i in files]

# the line below launches actual computations
results = dask.compute(delayeds)

# after computation is over the results object will 
# contain a list of pandas dataframes
dask_array = dd.from_delayed(delayeds) # here instead of pd.concat
dask_array.compute().to_csv(r"D:\XX\XX\XX\XX\XXX\*.csv") # Please be aware of the dtypes on your Excel.

Concatenate Excel Files using Dask

1 Answers