Efficiently extracting a few values from several large Pandas dataframes

user_12305295 0
Viewed 106

I have several (tens to hundreds) DataFrames that I'm reading from .tab/.csv files. The DFs are substantial (thousands of rows, hundreds of columns, tens of MBs), but I only need to pick out just a couple of values from each DataFrame.

This is straightforward enough to do by looping through the files and extracting the needed values from each:

values = []

for file in datalist:
    df = pd.read_csv(file)

    # Identify values by row/column labels as positions may change
    val0 = df.loc['row_index_0', 'column_number_0']
    val1 = df.loc['row_index_1', 'column_number_1']
    val2 = df.loc['row_index_2', 'column_number_2']

    values.append([val0, val1, val2])

But I'm wondering if I need to worry about doing it efficiently, and if so, how best to do that. Like, do I need to worry about a simple for loop like that creating a bunch of sizeable df objects that then float around taking up memory?

And if that's a potential problem, what can I do to make sure the df objects are purged once no longer needed? Is there a simple context manager approach like using with statements?

pandas dataframe python
1 Answers

You cannot avoid iterating through all the csv files, but you can make it more efficient by not reading the entirety of each file into a dataframe. If you are only interested in the first few (e.g. 3) rows and columns, then just read those as demonstrated here:

values = []

for file in datalist:
    df = pd.read_csv(file, nrows=3, usecols=[0,1,2])
    val0, val1, val2 = df.iloc[r0, c0], df.iloc[r1, c1], df.iloc[r2, c2]
    values.append([val0, val1, val2])
user_91620000
Related