How to concatenate multiple dataframes having different column names and length

user_2350145 0
Viewed 429

I am having multiple dataframes. All are having different column names and lengths. For example df1 has columns like ['c1', 'c2', 'c3'], df2 has columns like ['d1', 'd2', 'd3', 'd4'] and so on.

I want to concatenate all the dfs one under another. I don't care about the column name preservation. Resultant df will have all the values of df1 and df2 and so on.

Right now I'm doing pd.concat([df1, df2], axis=0) which is making the resultant df both df1 and df2 columns side by side. I want them one under another.

pandas dataframe python
2 Answers

If column names isn't important, we can also consider taking the numpy values of the dataframes by DataFrame.values and concat together using pd.concat(), like below:

pd.concat([pd.DataFrame(dfi.values) for dfi in [df1, df2]], ignore_index=True)
user_150706970

Since your DataFrames can have a different number of columns, rename the labels to be their integer position that way they align underneath for the join. The result will have an Int64Index on the columns, up to the length of the widest DataFrame you provide in the concat.

Sample Data

import pandas as pd
import numpy as np

np.random.seed(123)
df1 = pd.DataFrame(np.random.choice(['foo', 'bar'], (2, 3)),
                   columns=['c1', 'c2', 'c3'])
df2 = pd.DataFrame(np.random.randint(11, 20, (3, 4)),
                   columns=['d1', 'd2', 'd3', 'd4'])

Code

df = pd.concat([x.rename(columns=dict(zip(x.columns, range(x.shape[1])))) 
                for x in [df1, df2]],
               ignore_index=True) 

#     0    1    2     3
#0  foo  bar  foo   NaN
#1  foo  foo  foo   NaN
#2   17   12   14  17.0
#3   12   11   12  11.0
#4   11   14   15  11.0
user_43333590
Related