Read in multiple files using pd.read_csv() and save each file as a different variable

user_13566536 0
Viewed 530

I have a folder which contains multiple (over 50) .txt files which all have the same structure (same columns, but different lengths sometimes).

Is there a way to read these files using pd.read_csv() and save each file as a variable, say res1 = file1, res2 = file2 etc? I need to use python pandas to read the files.

I don't know if this is a good practice, but I'd like to avoid loading each file separately.

pandas dataframe python
2 Answers

You could loop over all the files using os.listdir():

res = [pd.read_csv(file) for file in os.listdir('your_directory')]

Each df will then be at its index. i.e res[0] will be df1

user_141636950

If these CSVs are large files, then keep in mind that when you read them in all at once, you could run out of memory.

If you don't have to process the different CSVs simultanously, you should probably process them after each other.

If you really need each file in a separate dataframe, and memory is sufficient for the task, just use a loop and store them in a list or dict, or in this case a list of dicts – it depends on what you want to do:

df_list = []
for f in files:
   df_list.append({
      "filename": f,
      "df": pd.read_csv(f)
   })

@ghandifloss answer could also serve your purpose well.

user_84120380
Related