I have a list of DataFrames [df1, df2, df3, df4, ...]
I want to do pd.concat, and only use the columns of a df it doesn't exist in previous dfs
dfs = [df1, df2, df3, df4, ...]
final_df = df1
for df in dfs[1:]:
final_df = pd.concat([final_df, df[df.columns.difference(final_df.columns)]])
Is there a better way than what I did above?
You can achieve this using sets. consider the following example:
s1 = set(('col1', 'col2'))
s2 = set(('col2', 'col3'))
s2.difference(s1) # this returns {'col3'}
So what you can do is accumulate a set of column names and for every new column append it to your data frame and the column name to the accumulated set.
On further reading of you question, it appears that you did pretty much the same thing. the only improvement that I can think of is if you have many frame and/or large frames then don't concat on every iteration because that creates a new frame and that will get slower and slower. Instead put all of them in a list and make a frame of them in the end.