Dynamic List of Dataframes

user_3768977 0
Viewed 31

We receive data in the form of emails containing csvs.

These emails are stored in an s3 bucket and via a python job we read each csv, create a standard dataframe structure, and concatenate all these dataframes into a single frame.

However, there are times in which we don't get the same number of csv's and due to the unique nature of each file, we have to create a separate dataframe for each csv and then concatenate together.

What I'd like to do is make the process more tolerant to the variable number of dataframes that will be created.

Current the code is as follows:

try:
   Read CSV
   Do stuff to prepare standard dataframe
except:
   print('fail')
...numerous try-catch blocks...
frames = [ALL CREATED DATAFRAMES]
result_frame = pandas.concat(frames)

What I need is code that will allow me to get only the dataframe objects that exist and use that as the list for frames.

I tried using dir() and creating a list from the resulting generator object but that is only the string variable name, not the variables themselves.

ideas?

python-3.x python
1 Answers

Answer!

I've named each dataframe variable as 'XXXXX_df'. Using globals() I was able to get the name and value of each and then append it.

var_global = globals()
var_global_keys = var_global.keys()
variables = list(x for x in var_global_keys if '_df' in x)
resultframe = pd.DataFrame()
for frame in variables:
    resultframe = resultframe.append(var_global[frame])
user_37689770
Related