Reading in a list of files into a list of DataFrames

user_5286312 0
Viewed 2503

I'm trying to read a list of files into a list of Pandas DataFrames in Python. However, the code below doesn't work.

files = [file1, file2, file3]

df1 = pd.DataFrame()
df2 = pd.DataFrame()
df3 = pd.DataFrame()

dfs = [df1, df2, df3]

# Read in data files
for file,df in zip(files, dfs):
    if file_exists(file):
        with open(file, 'rb') as in_file:
            df = pd.read_csv(in_file, low_memory=False)
            print df        #the file is getting read properly

print df1    #empty
print df2    #empty
print df3    #empty

How to I get the original DataFrames to update if I pass them into a for-loop as a list of DataFrames?

pandas dataframe list python
6 Answers

Custom-written Python function that appropriately handles both CSV & JSON files.

def generate_list_of_dfs(incoming_files):
    """
    Accepts a list of csv and json file/path names.
    Returns a list of DataFrames.
    """
    outgoing_files = []
    for filename in incoming_files:
        file_extension = filename.split('.')[1]
        if file_extension == 'json':
            with open(filename, mode='r') as incoming_file:
                outgoing_json = pd.DataFrame(json.load(incoming_file))
                outgoing_files.append(outgoing_json)
        if file_extension == 'csv':
            outgoing_csv = pd.read_csv(filename)
            outgoing_files.append(outgoing_csv)
    return outgoing_files

How to Call this Function

import pandas as pd
import json    
files_to_be_read = ['filename1.json', 'filename2.csv', 'filename3.json', 'filename4.csv']
dataframes_list = generate_list_of_dfs(files_to_be_read)
user_122799180

Here is a simple solution that avoids using a list to hold all the data frames, if you don't need them in a list.

import fnmatch

# get the CSV files only
files = fnmatch.filter(os.listdir('.'), '*.csv')
files

Output which is now a list of the names:

['Feedback Form Submissions 1.21-1.25.22.csv',
 'Feedback Form Submissions 1.21.22.csv',
 'Feedback Form Submissions 1.25-1.31.22.csv']

Now create a simple list of new names to make working with them easier:

# use a simple format
names = []
for i in range(0,len(files)):
    names.append('data' + str(i))
names

['data0', 'data1', 'data2']

You can use any list of names that you want. The next step take the file names and the list of names and then assign them to the names.

# i is the incrementor for the list of names
i = 0

# iterate through the file names
for file in files:
    # make an empty dataframe
    df = pd.DataFrame()
    # load the first file in
    df = pd.read_csv(file, low_memory=False)
    # get the first name from the list, this will be a string
    new_name = names[i]
    # assign the string to the variable and assign it to the dataframe 
    locals()[new_name] = df.copy()
    # increment the list of names
    i = i + 1

You now have 3 separate dataframes named data0, data1, data2, and do commands like

data2.info()
user_49507530
Related