How to append .pkl files in for loop to pandas dataframe created in for loop?

user_9536233 0
Viewed 1096

I have a seemingly simple piece of code but somehow it is not working. The goal of the code is to find all pickle data in a folder, load the first one in a for loop as a pandas dataframe which is named under a variable which did not exist before, if the variable exists, it should load the remaining pickle files as pandas and append them to the newly created pandas dataframe from the first loop:

import pandas as pd
import os

# Creating the first Dataframe using dictionary 
df1  = pd.DataFrame({"a":[1, 2, 3, 4], 
                         "b":[5, 6, 7, 8]}) 
  
# Creating the Second Dataframe using dictionary 
df2 = pd.DataFrame({"a":[1, 2, 3], 
                    "b":[5, 6, 7]}) 


df1.append(df2)

works fine printing:

    a   b
0   1   5
1   2   6
2   3   7
3   4   8
0   1   5
1   2   6
2   3   7

However when I try to append the dataframes from my stored pickle files in a for loop it does not print an error but it only works for the first dataframe:

df1.to_pickle("DF1.pkl")
df2.to_pickle("DF2.pkl")

files = [f for f in os.listdir('.') if os.path.isfile(f)]
#The line above should produce the line below
files=["DF1.pkl", "DF2.pkl"]

for i in files:
    if ".pkl" in i:
        if "ALL_DATA" not in globals():
            ALL_DATA=pd.read_pickle(i)
        else:
            ALL_DATA.append(pd.read_pickle(i))

which only prints:

a   b
0   1   5
1   2   6
2   3   7
3   4   8

Who can help me clarify?

pandas pickle append for-loop python
1 Answers

DataFrame.append returns a new object so though you call ALL_DATA.append(pd.read_pickle(i)) as you never write that back to ALL_DATA those changes are discarded. You need to assign the changes back:

ALL_DATA = ALL_DATA.append(pd.read_pickle(i))

However, appending in a loop is inefficient as it will copy data upon every iteration so you should avoid it. Instead, append to a list, which is fast, and then concat once after the loop.

l = [] # Holds everything you may possibly append
for i in files:
    if ".pkl" in i:
        if "ALL_DATA" not in globals():
            ALL_DATA=pd.read_pickle(i)
        else:
            l.append(pd.read_pickle(i)) # List append which modifies `l`

# Create df from ALL_DATA and everything that you append
ALL_DATA = pd.concat([ALL_DATA, *l])
user_43333590
Related