Appending Pickle Files in Python

user_6557659 0
Viewed 8855

I have 100 dataframes (formatted exactly the same) saved on my disk as 100 pickle files. These dataframes are each roughly 250,000 rows long. I want to save all 100 dataframes in 1 dataframe which I want to save on my disk as 1 pickle file.

This is what I am doing so far:

path = '/Users/srayan/Desktop/MyData/Pickle'
df = pd.DataFrame()
for filename in glob.glob(os.path.join(path, '*.pkl')):
    newDF = pd.read_pickle(filename)
    df = df.append(newDF)
df.to_pickle("/Users/srayan/Desktop/MyData/Pickle/MergedPickle.pkl")

I understand that pickle serializes the data frame but is it necessary for me to take my pickle file, unserialize it, append the data frame, and then serialize it again? Or is there a faster way to do this? With all the data I have, I am getting slowed down

pandas pickle dataframe python
2 Answers

A more compact version in one line:

df = pd.concat(map(pd.read_pickle, glob.glob(os.path.join(path, '*.pkl'))))
user_108630910
Related