Scaled/Optimized merging of multiple DataFrames in Python

user68W21 0
Viewed 23

I have a list of several DataFrames in Python that I am attempting to merge. Currently, I am merging them like so:

import pandas as pd
from functools import reduce

DATAFRAMES = [DF1, DF2, . . . , DFN]

FINAL = reduce(lambda  left,right: pd.merge(left, right, on = 'key_column_name',
                                            how = 'outer'), DATAFRAMES)

This solution works well for the merging aspect, however, my question is whether this is the most efficient way to do this for some large N DataFrames.

Is there a way to optimize/alter this code or perhaps use a different library like dask for when I have several large dataframes that need to be merged?

Thanks!

merge optimization dataframe pandas python
1 Answers

I would suggest using concat() method, so that pandas performs one operation under the hood instead of N-1.

FINAL = pd.concat(
    [df.set_index('key_column_name') for df in DATAFRAMES],
    axis=1,
).reset_index()
Boris Silantev0
Related